Robots.txt: Steering Crawlers and Saving Crawl Budget

What is the robots.txt?

The robots.txt is a simple text file in the root directory of a website (yourdomain.com/robots.txt), which gives search engine crawlers instructions on which areas they may search and which not. It follows the Robots Exclusion Protocol (REP) and is the first file a bot retrieves when visiting a website.

Important right up front: the robots.txt controls crawling, not indexing. It is not a mechanism to reliably keep pages out of Google’s results. A page blocked via robots.txt can still be indexed if other websites link to it – then it usually appears without a description in the index. For a secure exclusion you need other methods, which we describe under removing URLs from Google.

The robots.txt is one of the core tools in technical SEO. Used correctly, it steers crawlers to your important content and saves your crawl budget.

Why the robots.txt is important

A correctly configured robots.txt offers several advantages:

• Set crawling priorities: You steer bots to the truly important content and exclude irrelevant or duplicate areas. Especially relevant for large websites with many URLs.
• Optimize crawl budget: Search engines crawl only a limited number of URLs per period. Whoever excludes unimportant areas uses this budget more efficiently.
• Save server resources: Fewer unnecessary bot requests mean less server load.
• Support sitemap use: A reference to the XML sitemap helps crawlers find all important URLs.

Structure and syntax

A robots.txt consists of one or more groups of directives. Each group begins with a User-agent line, followed by Disallow or Allow instructions. The most important components:

User-agent – which crawler the rules apply to. * applies to all bots, Googlebot only to Google’s main crawler.

Disallow – forbids the crawling of certain paths. The path is relative to the root directory.

Allow – deliberately allows a URL, even if a superordinate directory is blocked. Supported above all by Google and Bing. In case of conflicts, the more specific rule (the longer path) wins here.

Sitemap – specifies the full path to the XML sitemap.

A typical example for a WordPress site:

Special characters

• **Wildcard (*)** stands as a placeholder for any character string. Disallow: /*.pdf$ blocks all URLs that end in .pdf.
• Dollar sign (`$`) marks the end of the line. Disallow: /directory$ blocks /directory, but not /directory/subpage.
• Hash (`#`) introduces a comment that crawlers ignore.

Best practices

So that your robots.txt works cleanly, keep these points in mind:

• Correct name and location: The file must be named exactly robots.txt and lie in the root directory of the domain. It is saved as UTF-8, ideally without a byte order mark (BOM).
• Do not block CSS and JavaScript: Modern pages need these resources so that Google can render them correctly. If you block them, Google may misunderstand the page.
• Use the Sitemap directive: Always reference your XML sitemap.
• One file per (sub)domain and protocol: Every subdomain and every protocol (http/https) needs its own robots.txt.
• `noindex` does not belong in the robots.txt: This unofficial directive has been fully ignored by Google since September 2019. For an indexing exclusion, use the meta robots tag or the X-Robots-Tag HTTP header.
• Check regularly: Especially after relaunches or server migrations. A too-broad Disallow: / from the test environment can accidentally block the entire website.

Limits and alternatives

The robots.txt has clear limitations that you should know:

• No guarantee of compliance: Reputable crawlers stick to it, malicious bots and scrapers often do not. The robots.txt is not a security mechanism.
• Blocked pages can be indexed: If other pages link to a blocked URL, Google can still add it to the index.
• Caching: Google usually caches the robots.txt for up to 24 hours. Changes therefore do not take effect immediately.
• Maximum size: Google processes up to 500 KiB; anything beyond that can be ignored.

For tasks that go beyond crawling management, there are more suitable tools: the meta robots tag (<meta name="robots" content="noindex">) for the reliable exclusion of individual HTML pages from the index, the X-Robots-Tag in the HTTP header for non-HTML files like PDFs, and password protection for truly sensitive areas.

Robots.txt and AI crawlers

With the rise of AI models, new crawlers have appeared that collect content for training data or AI answers. Many have their own user agents, for example GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, or Google-Extended (controls the use by Google’s AI products). These can be steered deliberately:

Whether every AI bot respects these instructions is not guaranteed – but it has become common practice. Note the trade-off: whoever locks out all AI crawlers may reduce their chances of being cited as a source in AI search systems. This decision belongs to strategy, not only to technology.

Testing the robots.txt

The former robots.txt tester in the Google Search Console was shut down at the end of 2023. Here is how to check your file today:

• robots.txt report in the Search Console: Under the settings, Google shows which robots.txt was last retrieved, when, and whether errors occurred.
• URL inspection: For individual URLs, the URL Inspection tool shows whether the Googlebot may crawl them or is blocked.
• External validators: Various SEO tools offer their own robots.txt testers, with which you can simulate rules against specific URLs.

After every change you should test whether the rules take effect as intended – a single typo can block important areas or open up unwanted ones.

Conclusion

The robots.txt is a fundamental tool for controlling how crawlers explore your website. Used correctly, it improves crawl efficiency and saves server resources. What is decisive is to know its limits: it controls crawling, not indexing, and is not a security measure. With clean syntax, regular tests, and the right tool for the respective purpose, it becomes a reliable building block of your technical SEO.