Look up GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers. Copy a User-agent robots.txt snippet per bot — no email required.
| Bot | Vendor | Purpose | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training and retrieval crawler for ChatGPT | |
| ChatGPT-User | OpenAI | User-initiated browsing from ChatGPT | |
| OAI-SearchBot | OpenAI | Search indexing for ChatGPT search | |
| ClaudeBot | Anthropic | Training and retrieval crawler for Claude | |
| Claude-User | Anthropic | User-initiated browsing from Claude | |
| Claude-SearchBot | Anthropic | Search indexing for Claude | |
| PerplexityBot | Perplexity | Indexing crawler for Perplexity answers | |
| Perplexity-User | Perplexity | Live retrieval from Perplexity | |
| Google-Extended | Gemini / AI Overviews extended crawl | ||
| Googlebot-Extended | Alternate Google AI Overviews user-agent | ||
| Applebot-Extended | Apple | Apple Intelligence / Applebot AI features | |
| Amazonbot | Amazon | Amazon Alexa and product graph crawl | |
| Bytespider | ByteDance | TikTok / ByteDance AI crawl | |
| cohere-ai | Cohere | Cohere model retrieval crawler | |
| meta-externalagent | Meta | Meta AI crawler | |
| DuckAssistBot | DuckDuckGo | DuckDuckGo AI Assist crawler |
GPTBot is OpenAI’s crawler. Allowing User-agent: GPTBot in robots.txt lets ChatGPT’s retrieval and training pipelines read public pages you have not disallowed.
Unless you have a legal reason to block training, allow the major assistants (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Blocking them cedes AI answers to competitors who stay crawlable.
No. Googlebot is classic search. Google-Extended / Googlebot-Extended is the AI Overviews and Gemini path. Blocking Extended does not hide you from web search, but it can hide you from AI Overviews.
Copy the User-agent block and merge it into /robots.txt. Prefer Allow: / for public content. Re-test with the AI Robots.txt Auditor.
Yes. It is ungated reference material — search, filter, and copy without giving an email.
These tools fix one GEO signal. A BrandCitation scan shows whether ChatGPT, Perplexity, Gemini, and Claude mention your brand — and who they recommend instead.