Controlling GPTBot and AI Bots via robots.txt

The control in one sentence

You steer the access of AI crawlers primarily via the robots.txt – a text file in the root directory of your domain in which you define per bot which areas it may retrieve. For each AI bot you specify a rule with its user-agent name and thereby deliberately allow or block access.

This article covers the practical implementation for AI bots. It assumes the basics of robots.txt – syntax, structure, pitfalls; those are described in detail in the article robots.txt. Anyone who wants to know which bots exist at all should first read Understanding AI crawlers and bots.

What a rule for an AI bot looks like

A robots.txt rule consists of a User-agent line that names the bot, and one or more Disallow or Allow lines. If you want, for example, to completely lock out OpenAI’s training crawler:

Conversely, you grant it full access with an empty Disallow line:

You can also block only certain areas – for example a paid members’ area, while the rest stays open:

Addressing the most important AI bots specifically

Every provider has its own user-agent names. This rule, for instance, excludes the most common training and data-collection crawlers from your site all at once:

For Google there is a special case: access for AI training runs via the token Google-Extended. With it you prevent the training use without endangering your normal Google visibility and the AI Overviews:

Important: Google-Extended controls only the AI training use. The regular Googlebot – and thus your classic ranking as well as the AI Overviews – remains unaffected by it.

Steering training and search visibility separately

The decisive trick for GEO: many providers separate their bots by purpose, and you can exploit this separation. At OpenAI, for example, there is GPTBot (training), OAI-SearchBot (indexing for ChatGPT Search) and ChatGPT-User (live retrieval on a user request).

If you do not want your content to flow into model training, but very much want it cited in ChatGPT answers, you deliberately block only the training crawler:

This way you protect your content from being used for training, but stay visible for live citation. Exactly this differentiated control is often the smarter choice than a blanket yes or no. The exact bot names do change, however – check them in the respective provider’s official documentation before rolling out.

The trade-off: visibility versus content protection

Whether you allow or block AI bots is a strategic decision with two poles:

• Visibility argues for access. Anyone who locks out AI crawlers can hardly be cited in the respective AI answers. For retrieval and search bots, blocking almost always means: invisible in this AI.
• Control argues against access. Some brands do not want their unique content to flow into training and be reproduced without compensation – for example in the case of elaborately created expert content or journalistic works.

The pragmatic line for most website operators who bet on GEO: allow search and retrieval bots (they bring visibility), and deliberately weigh training bots. A blanket sweep that blocks all AI bots reliably costs you reach in AI search.

Important: robots.txt is a request, not a block

A common misconception: robots.txt blocks nothing technically. It is a polite instruction that reputable providers adhere to – GPTBot, Google-Extended, PerplexityBot and the other major bots respect the rules. But a disreputable scraper can simply ignore them.

If you really have to protect content technically, you need harder means: access protection via login, server-side blocks of certain IP ranges, or a firewall rule. robots.txt is the right tool to communicate your intention to cooperating AI bots – no more, but no less.

And one more point: allowing access is only half the battle. The bot must also be able to read your content. Why many AI crawlers fail at JavaScript and how you avoid it is covered in the article Rendering and Accessibility for AI Crawlers. All the technical steps are bundled in the technical GEO base checklist.

Conclusion

Via robots.txt you deliberately steer which AI bots may retrieve your content – with a User-agent rule per bot and Disallow or Allow. The most important lever for GEO is the separation by purpose: you can block training crawlers like GPTBot or the token Google-Extended, while you allow search and live-retrieval bots and thus stay visible. Weigh visibility against content protection – a blanket block costs reach. And keep in mind: robots.txt is a request to cooperating bots, not a technical barrier. For real protection you need server-side means.