AI crawlers and robots.txt: who visits your site and how to control it
You control AI crawler access mainly through robots.txt — a public text file in your site’s root that hands crawlers the rules for crawling. This guide shows you how to set up robots.txt for AI crawlers: which crawlers are visiting sites today, how to allow them so you stay citable, how to restrict their use of your content for training, and where robots.txt stops being enough.
Don’t confuse this with turning off Google AI Overviews in the results (that has its own separate guide). This is about crawler access to your site across AI services.
Which AI crawlers and control tokens robots.txt deals with
The big players run several crawlers with different jobs — one for training, one for search, one for fetching a page when a user asks about it:
| Crawler | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | model training |
| OAI-SearchBot | OpenAI | search and citations in ChatGPT |
| ChatGPT-User | OpenAI | fetching a page on a user’s request |
| ClaudeBot | Anthropic | model training |
| Claude-SearchBot | Anthropic | search |
| Claude-User | Anthropic | on-demand fetching |
| PerplexityBot | Perplexity | crawling for indexing and answers (per the operator’s documentation) |
| Google-Extended | control token for using content in Google’s generative AI services, e.g. Gemini (not classic indexing) | |
| CCBot | Common Crawl | building a public dataset of the web; the data can also be used for model training |
The specific names change — check each service’s documentation for the current list.
Want more citations: let the search crawlers through
If you want AI tools to find you and cite you more often, let the search crawlers through above all. In robots.txt in your site’s root:
# Allow AI crawlers for search and citations
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /When no rule disallows a crawler, it usually has access even without an explicit Allow — listing it is mainly for clarity. But allowing crawlers doesn’t earn you citations on its own; it only removes the technical obstacle, so your content can reach the search and answer layers. This layer builds on SEO for ChatGPT and belongs in a broader AI SEO audit.
Don’t want your content in training data: restrict the training crawlers
If you want to signal that you’d rather not have your content used for model training, restrict the training crawlers (this is a signal to legitimate operators, not an enforceable guarantee):
# Restrict crawlers used for model training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /Keep the training crawlers and the search crawlers straight as you do it: you can disallow training (GPTBot) while still letting search through (OAI-SearchBot), so you stay citable. Blocking everything at once sacrifices your citability.
Handle Google-Extended separately
Google-Extended is a common source of confusion. It isn’t a classic indexing crawler but a control token that, per Google’s documentation, governs whether your site’s content can be used for Google’s generative AI services (Gemini Apps and Vertex AI, for example). According to that documentation, blocking it doesn’t control regular crawling, indexing, or rankings in Google Search — classic Googlebot handles that. So you can disallow Google-Extended without directly changing your visibility in Google Search.
Robots.txt is a request, not a lock
This is the most important thing to take away: robots.txt is a voluntary request, not a technical barrier. Well-behaved crawlers respect the rules, but it isn’t enforceable:
- Some crawlers ignore the rules — Bytespider, for example, ranked among the most active AI crawlers in widely cited 2024 data and often failed to honor robots.txt.
- For harder blocking, act at the server, firewall, or CDN level. A user agent can be spoofed, so pair it with IP range verification, rate limiting, and log monitoring.
- Don’t put sensitive or non-public URLs in robots.txt as a substitute for security — the file is public, and
Disallowdoesn’t stop anyone from opening the page directly.
Treat robots.txt as the first layer for well-behaved crawlers, not as a guarantee.
How to verify your setup
Once you’ve edited robots.txt, check the following:
Robots.txt setup check
- The file is reachable open https://yourdomain.com/robots.txt — it has to return HTTP 200 and contain your rules.
- The right rule group applies most parsers give a specific User-agent precedence over a generic User-agent: *; test the behavior.
- Watch your server logs once it's live, confirm the crawlers behave the way you expect.
- Keep an eye on hard blocks where you block, watch for 403/429 responses and crawlers blocked by mistake.
What about llms.txt?
The llms.txt file comes up often. Take it with a grain of salt: it’s a proposed format that describes your site’s content for AI, not an access-control mechanism. Based on the information available, the major crawlers don’t reliably use it as of 2026. So for access control, rely on robots.txt and your server, not on llms.txt.
Common mistakes
Blocking every AI crawler
A blanket block takes out the search crawlers too, so AI tools stop citing you.
Fix: Let the search crawlers through at least.
Blocking Google-Extended over rankings
Google-Extended only affects Google’s generative AI services, not regular search.
Fix: It has no effect on Google Search rankings.
Treating robots.txt as a lock
Robots.txt won’t stop the crawlers that don’t play by the rules.
Fix: Handle hard blocking on your server or firewall.
Mistaking llms.txt for access control
llms.txt isn’t a blocking tool.
Fix: Use robots.txt and your server to block.
What’s next: set this up across the whole site
Crawler access is only one layer of AI visibility. For how to build on it with structure, structured data, and citable content, see the practical SEO for AI checklist.
