Robots.txt

By Sheriff Docs ~ 7 min read

A robots.txt file tells search engine crawlers how they should interact with your website.

Although the name sounds intimidating, robots.txt is simply a small text file placed in the root of your website. Search engines such as Google read this file before crawling your pages.

Sheriff automatically generates this file for every portal that has a configured site_url.

Why Is It Important?

The robots.txt file serves two primary purposes:

It tells search engines they are allowed to crawl your website.
It tells search engines where your sitemap is located.

For most Sheriff sites, you will never need to edit this file yourself.

Generated File

When you build your portal, Sheriff automatically creates:

dist/
└── sheriff/
    └── robots.txt

dist/
└── sheriff/
    └── robots.txt

The file is generated alongside your sitemap.xml.

Example

A typical generated robots.txt file looks like this:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

The Sitemap entry is generated automatically from the site_url configured in your portal's config.yall file.

User-agent

A User-agent identifies a search engine crawler.

For example:

User-agent: Googlebot

User-agent: Googlebot

would apply only to Google's crawler.

Sheriff generates:

User-agent: *

User-agent: *

The * means "all search engine crawlers."

Allow

The Allow directive tells crawlers which parts of your website they are permitted to crawl.

Sheriff generates:

Allow: /

Allow: /

The / represents the root of your website, allowing search engines to crawl your entire site.

Sitemap

The Sitemap directive tells search engines where to find your sitemap.

For example:

Sitemap: https://example.com/sitemap.xml

Sitemap: https://example.com/sitemap.xml

This allows crawlers to quickly discover every page on your website.

Modules Involved

Trailboss — generates routes.json, sitemap.xml, and robots.txt.