Reference
Tracked AI crawlers
The seven crawlers the checker tests robots.txt against, and why each matters.
The AI_CRAWLERS export (lib.js#L6-L14) is the exact list the
crawler-access check tests against, and the list generate_ai_readiness_fixes
welcomes in its robots.txt template:
| User-agent | Feeds |
|---|---|
| GPTBot | ChatGPT (OpenAI index and training) |
| OAI-SearchBot | ChatGPT Search |
| ClaudeBot | Claude |
| PerplexityBot | Perplexity |
| Google-Extended | Google Gemini and AI Overviews |
| CCBot | Common Crawl (feeds many LLMs) |
| Bingbot | Bing (powers ChatGPT Search results) |
How blocking is evaluated
The robots.txt parser groups rules by User-agent and picks the most specific
group for each crawler (exact name first, then *). A crawler counts as blocked
only when its group carries Disallow: / or Disallow: /* without a root-level
Allow (lib.js#L47-L64). Blocking some but not all tracked
crawlers yields warn; blocking all seven yields fail.
