AI crawlers and robots.txt: what you can control

Three kinds of AI bots

Not every AI bot does the same job. Training crawlers collect content that may be used to train AI models. Search crawlers build an index from which AI search features answer questions and link to sources. User fetchers open a page only when a user asks something that needs it. The large operators use a separate token for each job, so you can decide per job. OpenAI, for example, writes that its settings are independent: you can block training and still appear in its search.

Which tokens exist

As of 30 October 2026, the operators describe their bots like this:

• OpenAI: GPTBot collects content that may be used to train its foundation models, OAI-SearchBot is used for the search features of ChatGPT, and ChatGPT-User visits pages for actions that a user starts. OpenAI writes that robots.txt rules may not apply to such actions.
• Anthropic: ClaudeBot collects content that could help train its models, Claude-SearchBot improves search results, and Claude-User fetches pages when a question of a user requires it. According to Anthropic, all three respect robots.txt.
• Perplexity: PerplexityBot shows and links websites in Perplexity search results and, according to Perplexity, is not used to crawl content for AI foundation models. Perplexity-User fetches pages when a user asks and, as Perplexity writes, generally ignores robots.txt.
• Google: Google-Extended is not a separate crawler but a token in robots.txt. It controls whether content that Google crawls may be used to train future Gemini models and for grounding in Gemini. According to Google, it does not affect inclusion or ranking in Google Search.
• Apple: Applebot-Extended does not crawl pages either. It controls whether data crawled by Applebot may be used to train Apple's generative AI models. Pages that block it can still appear in Apple's search results.

Other operators, such as Common Crawl, Meta, Amazon and DuckDuckGo, publish their own tokens. Names change, so check the documentation of the operator.

How the rules work

• The file lives at the root of the host, such as https://example.com/robots.txt, and applies only to that protocol, host and port.
• Rules stand in groups under User-agent lines. According to Google, a crawler follows only the most specific group that matches it. If a group names it, the group for all crawlers (*) no longer applies to it.
• Agent names are not case-sensitive. Within a group, the longest matching rule wins; if allow and disallow are equally specific, Google uses allow.
• Google reads at most 500 KiB of the file. If robots.txt answers with a 4xx status other than 429, Google treats the site as if there were no file, so everything may be crawled.

User-agent: *
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

In this example, the training bots of three operators may not fetch anything, the search crawlers may fetch everything except /admin/, and all other crawlers follow the first group. The search crawlers repeat Disallow: /admin/ because the rules for * no longer apply to them.

What blocking means for your visibility

Each group affects a different kind of visibility. If you block a training crawler, you ask its operator not to use your content for training; Anthropic, for example, describes blocking ClaudeBot as a signal for future material. If you block a search crawler, your pages may not appear in the AI search of that operator. OpenAI writes that sites that block OAI-SearchBot are not shown in ChatGPT search answers, but may still appear as navigational links. If you block user fetchers, assistants that respect the rule cannot open your page when someone asks about it.

Google is a special case. Blocking Google-Extended does not remove your site from Google Search. AI Overviews and AI Mode are part of Google Search: according to Google, a page must be indexed and eligible to be shown with a snippet, and no special files or markup are needed. To limit what Google shows from a page, Google names nosnippet, data-nosnippet, max-snippet and noindex. These apply to Search as a whole, not only to AI features.

robots.txt and llms.txt

The two files do different jobs. robots.txt says which automated access you accept. llms.txt is a proposal for a Markdown file that gives AI assistants a short overview of your most important pages, mainly at the moment someone asks a question; our guide on llms.txt explains the format. According to llmstxt.org, llms.txt can complement robots.txt by giving context for content that is allowed, but it does not replace it. Decide access in robots.txt first.

A sensible order

1. 1Decide per group what you want: training, AI search and user fetches. One possible choice: block training, allow search and user fetches.
2. 2Read your current robots.txt: which groups exist and what the group for all crawlers blocks.
3. 3Add a group for each decision and repeat the rules from the * group that should still apply.
4. 4After publishing, open /robots.txt in the browser and check the file.
5. 5Review your list from time to time. Operators add, rename and retire bots.

Build your rules with Do4U

The AI crawlers in robots.txt tool keeps a list of AI crawlers with operator, purpose and a link to the documentation, in three groups: model training, search, and fetches on a request of a user. You choose per group or per crawler: allow, block or leave unchanged, optionally for selected paths. If you enter your website, Do4U reads your robots.txt, shows for each AI crawler the state before and after, and adds a marked AI section at the end. Your other groups and sitemap lines stay as they are. Fetchers whose operators say they may ignore robots.txt are marked. The code then checks the new file again. It runs without AI and is free.

Do4U builds the draft, you check and decide. robots.txt remains a request: whether a crawler follows it, Do4U cannot check.