Which AI Bots to Allow in robots.txt
A practical guide to AI crawler user-agents — GPTBot, Google-Extended, PerplexityBot, ClaudeBot — and how to configure robots.txt without blocking AI visibility.
If your robots.txt blocks AI crawlers with Disallow: /, you are actively suppressing AI search visibility. AI engines cannot learn about your brand if they cannot crawl your content.
BrandCitation's GEO audit awards 15 points when AI bots are not blocked — making this one of the highest-impact technical fixes.
AI crawler user-agents
These are the primary AI-related bots to know:
| User-agent | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Training and browsing for ChatGPT |
ChatGPT-User | OpenAI | ChatGPT browsing sessions |
Google-Extended | Gemini and AI product training | |
anthropic-ai | Anthropic | Claude training |
Claude-Web | Anthropic | Claude web browsing |
ClaudeBot | Anthropic | Claude crawling |
PerplexityBot | Perplexity | Perplexity search indexing |
YouBot | You.com | You.com AI search |
cohere-ai | Cohere | Cohere model training |
What "blocked" means
In robots.txt, a block looks like:
User-agent: GPTBot
Disallow: /
Disallow: / means the crawler cannot access any page on your site. If you have this for AI bots, remove it unless you have a specific legal requirement.
Recommended robots.txt for GEO
Allow general crawlers and explicitly allow AI bots:
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Adjust Disallow paths for your app's private routes (dashboard, admin, API).
Next.js example
BrandCitation uses Next.js robots.ts to generate robots.txt dynamically, allowing AI bots while blocking /dashboard/, /admin/, and /api/.
Should you block AI crawlers?
Allow them if you want AI engines to mention and cite your brand.
Consider blocking if you have strict IP/licensing concerns about training data — but understand this directly reduces AI visibility.
Most commercial brands optimizing for discovery should allow AI crawlers.
Combine with llms.txt
Allowing crawlers is step one. Step two: add /llms.txt so crawlers know what your site is about.
Audit your configuration
Order a full audit to check robots.txt, llms.txt, sitemap, schema, and AI visibility scores together — or start with the GEO checklist.
FAQ
Which AI bots should I allow in robots.txt? Allow GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, anthropic-ai, PerplexityBot, and Applebot-Extended if you want AI engines to discover and cite your content. Block only if you have a specific IP or licensing concern.
Will blocking GPTBot hide me from ChatGPT? Yes for browsing and partially for newer training data. ChatGPT's browsing layer relies on crawlable content, and blocking GPTBot removes a key discovery path — even if some of your content is already in model training.
What is Google-Extended? Google-Extended is a separate product token from Googlebot. Blocking it stops your content from being used for Gemini and Google's other AI products, but it does not affect your Google search ranking.
Do I need to block AI bots to protect my content? Only if you have a clear legal or licensing reason — paywalled content, original research, or copyright-sensitive material. For most commercial brands, blocking AI bots directly reduces AI visibility with no upside.
How do I verify my robots.txt is correct?
Visit https://yourdomain.com/robots.txt in a browser. Confirm AI bot user-agents are listed with Allow: / and that no Disallow: / rule overrides them.