generate-robotstxt

Awesome generator robots.txt.

Installation

npm install --save-dev generate-robotstxt

Usage

import robotstxt from "generate-robotstxt";

robotstxt({
  policy: [
    {
      userAgent: "Googlebot",
      allow: "/",
      disallow: "/search",
      crawlDelay: 2,
    },
    {
      userAgent: "OtherBot",
      allow: ["/allow-for-all-bots", "/allow-only-for-other-bot"],
      disallow: ["/admin", "/login"],
      crawlDelay: 2,
    },
    {
      userAgent: "*",
      allow: "/",
      disallow: "/search",
      crawlDelay: 10,
      cleanParam: "ref /articles/",
    },
  ],
  sitemap: "http://example.com/sitemap.xml",
  host: "http://example.com",
})
  .then((content) => {
    console.log(content);

    return content;
  })
  .catch((error) => {
    throw error;
  });

File based configuration

robots-txt.config.js

module.exports = {
  policy: [
    {
      userAgent: "Googlebot",
      allow: "/",
      disallow: ["/search"],
      crawlDelay: 2,
    },
    {
      userAgent: "OtherBot",
      allow: ["/allow-for-all-bots", "/allow-only-for-other-bot"],
      disallow: ["/admin", "/login"],
      crawlDelay: 2,
    },
    {
      userAgent: "*",
      allow: "/",
      disallow: "/search",
      crawlDelay: 10,
      cleanParam: "ref /articles/",
    },
  ],
  sitemap: "http://example.com/sitemap.xml",
  host: "http://example.com",
};

CLI

Awesome generator robots.txt

  Usage generate-robotstxt [options] <dest>

  Options:
     --config  Path to a specific configuration file.

Contribution

Feel free to push your code if you agree with publishing under the MIT license.

Changelog

License

Changelog

All notable changes to this project will be documented in this file. See standard-version for commit guidelines.

8.0.3 (2020-06-12)

Chore

• update deps

8.0.2 (2020-03-24)

Chore

• update deps

8.0.1 (2020-02-19)

Chore

• update deps

8.0.0 (2019-07-03)

Breaking change

• minimum require Node.js version is 10.13.0.

7.1.0 (2019-07-03)

Features

• cjs export (#123) (9c0507b)

7.0.0 - 2019-07-03

• Changed: minimum require Node.js version is 8.9.0.

6.0.1 - 2019-03-26

• Chore: migrate on ip-regex to avoid lodash vulnerable libraries

6.0.0 - 2019-01-09

• Chore: minimum required node-fs version is now ^7.0.1.

BREAKING CHANGE

• Chore: minimum required nodejs version is now >= 6.9.0.

5.0.7 - 2018-05-22

• Chore: minimum required cosmiconfig version is now ^5.0.0.

5.0.6 - 2018-05-05

• Fix: throw error when policy option is null.
• Chore: minimum required meow version is now ^5.0.0.
• Chore: minimum required node-fs version is now ^6.0.0.

5.0.5 - 2018-03-28

• Fix: allow empty values Disallow directive (according official spec).
• Fix: don't generate empty Clean-param.

5.0.4 - 2018-01-16

• Fix: typo in source code message errors.

5.0.3 - 2018-01-16

• Chore: minimum required cosmiconfig version is now ^4.0.0.

5.0.2 - 2017-12-12

• Chore: minimum required node-fs version is now ^5.0.0.

5.0.1 - 2017-11-28

• Chore: minimum required meow version is now ^4.0.0.

5.0.0 - 2017-11-15

• Changed: use `cosmiconfig for loading configuration.
• Feature: in CLI if the parent directory does not exist when you write robots.txt, it's created.

4.0.4 - 2017-10-09

• Chore: update deps.

4.0.3 - 2017-03-13

• Fixed: is-absolute-url package semver.

4.0.2 - 2016-12-30

• Fixed: host options is now processed based URL.
• Fixed: thrown error if the host option being IP address.
• Fixed: clarified error message on multiple and not string the userAgent option.
• Fixed: Host directive is now not containing 80 port.
• Fixed: thrown error if the cleanParam not string or array and if string not more than 500 characters.
• Fixed: supported unicode characters in a Allow and a Disallow directives.
• Fixed: thrown error if the sitemap option not an array or a string and not an absolute URL.

4.0.1 - 2016-10-27

• Chore: added CI test on node.js version 7.
• Documentation: improve README.md and fix typos.

4.0.0

• Added: crawlDelay to each police item.
• Added: cleanParam to each police item (used only Yandex bot).
• Chore: used remark-preset-lint-itgalaxy preset.
• Chore: updated devDependencies.
• Chore: updated copyright year in LICENSE.
• Chore: improved tests.
• Fixed: strict order directives for each User-agent.
• Fixed: added newline after each User-agent.
• Removed: crawlDelay from options.
• Removed: cleanParam from options.