SameDayDesk guide

AI crawler list 2026: the robots.txt tokens each operator publishes

As of the vendor pages checked on 2026-09-23, search answers and model training are different robots.txt tokens. OpenAI uses OAI-SearchBot for ChatGPT search answers and GPTBot for training. Anthropic's 7 April 2026 notice uses Claude-SearchBot for search indexing, ClaudeBot for training, and Claude-User for user-triggered retrieval. Perplexity uses PerplexityBot for search results, not for foundation-model training, and Perplexity-User for user actions. Google uses Googlebot for Google Search, including its AI features. Google-Extended is not a separate crawler and does not control inclusion in Google Search.

Last checked: 2026-09-23.

The list

Each token is its own group. Allowing one does not allow the others. RFC 9309 (September 2022) says these rules are not access authorization. A Disallow is a request to a crawler that chooses to honor it.

Token	Operator and role	Source
OAI-SearchBot	OpenAI. Surfaces sites in ChatGPT search features. Opting out means the site will not be shown in ChatGPT search answers. It can still appear as a navigational link.	OpenAI crawlers, checked 2026-09-23
GPTBot	OpenAI. Crawls content that may train generative AI foundation models. Disallow is the training opt-out. It is independent of OAI-SearchBot.	Same OpenAI page
ChatGPT-User	OpenAI. User-triggered visits from ChatGPT or a custom GPT. Not an automatic crawl. robots.txt may not apply. Not used to decide Search appearance.	Same OpenAI page
ClaudeBot	Anthropic. Collects content that may contribute to model training. Restricting it signals that future material should be excluded from training sets.	Anthropic crawler notice, 2026-04-07
Claude-SearchBot	Anthropic. Indexes content to improve search responses. Disabling it may reduce visibility in those search results.	Same notice, 2026-04-07
Claude-User	Anthropic. Retrieves pages when a person asks Claude. Disabling it may reduce visibility for user-directed web search.	Same notice, 2026-04-07
PerplexityBot	Perplexity. Surfaces and links sites in Perplexity search results. Not used to crawl for foundation models. A Disallow stops full or partial text indexing. A domain, headline, and short factual summary may still be indexed.	Perplexity crawlers and help center, modified 2026-09-14
Perplexity-User	Perplexity. User-triggered fetches. Generally ignores robots.txt. Not a training crawl.	Perplexity crawler docs
Googlebot	Google. Crawls for Google Search, including Discover and all Search features. AI Overviews and AI Mode are Search features. The robots.txt control for them is Googlebot, plus snippet and noindex controls.	Crawler list, updated 2026-07-14 and AI features, updated 2025-12-10
Google-Extended	Google. A robots.txt product token, not a separate HTTP user agent. Controls use of crawled content for future Gemini training and for grounding in Gemini apps and Vertex AI. Does not affect Google Search inclusion and is not a Search ranking signal.	Crawler list, updated 2026-07-14

A robots.txt block that matches those docs

This allows the automatic search and training tokens above. It does not control ChatGPT-User or Perplexity-User, because those operators say user-triggered fetches may ignore robots.txt. It also does not remove you from Google AI Overviews: that follows Googlebot.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

Put the file at /robots.txt on each host you care about. OpenAI says a robots.txt change can take about 24 hours to affect search. Anthropic says to repeat an opt-out on every subdomain, and that IP blocking alone may not persist because the crawler still needs to read robots.txt. Perplexity publishes current IP ranges at the JSON URLs on its crawler page. Google says the HTTP user-agent string can be spoofed, and it documents how to verify a real Google crawler.

Tokens this list leaves out on purpose

OpenAI also documents OAI-AdsBot for ad landing pages. It is not the Search control, so it is not in the block above. Other companies publish crawlers this page does not copy until their own page is checked the same way. Do not add a token because a roundup blog listed it.

Google's generative AI guide, last updated 2026-07-10, says you do not need an llms.txt file for Google Search, including its generative AI features. Google Search ignores it. Keeping one neither helps nor hurts Google rankings. Structured data is not required for those features.