Important: This documentation covers Yarn 1 (Classic).

For Yarn 2+ docs and migration guide, see yarnpkg.com.

keywords: robots.txt

found 96 packages in 14ms

robots-parser6.4mMIT3.0.1

A specification compliant robots.txt parser with wildcard (*) matching support.

samclarkeover 2 years agorobots.txt, parser, user-agent, scraper

generate-robotstxt484.7kMIT8.0.3

Awesome generator robots.txt

itgalaxyover 3 years agorobotstxt, robots.txt, generate, robots-txt

sitemapper133kMIT4.0.2

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

seantomburke5 months agoparse, sitemap, xml, robots.txt

npmGitHubHomepage

express-robots-txt111.3kMIT1.0.0

Express middleware to serve and generate robots.txt

modoscover 3 years agoexpress, robots, robots.txt, robots

robot-directives64.9kMIT0.4.0

Parse robot directives within HTML meta and/or HTTP headers.

stevenvachonover 3 years agocrawlers, header, html, http

Packages

Lightweight debugging utility for Node.js and the browser

debug, log, debugger

Lodash modular utilities.

modules, stdlib, util

React is a JavaScript library for building user interfaces.

A querystring parser that supports nesting and arrays, with a depth limit

querystring, qs, query, url

Higher-order functions and common patterns for asynchronous code

async, callback, module, utility

babel babel-core

Babel compiler core.

6to5, babel, classes, const