AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and Whether to Block Them

The short answer

AI crawlers are the bots AI companies send to read web pages, and they do three different jobs. Training crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot collect content for future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI answers can cite and link to them. User-triggered fetchers such as ChatGPT-User and Perplexity-User load a page live because someone asked about it. Each has its own robots.txt token, so blocking AI isn't all-or-nothing: the common middle ground is to disallow the training bots and allow the search bots, which opts you out of training while keeping you citable in AI answers. Just remember robots.txt is a request, not a lock.

The AI crawler list, grouped by purpose

Every token below comes from the operator's own documentation, checked in September 2026, and the last column reports what the operator itself says. Two entries aren't crawlers at all: Google-Extended and Applebot-Extended are robots.txt tokens that control how content fetched by Googlebot and Applebot may be used, so they never appear in server logs.

What blocking each group actually costs

Blocking training crawlers is the low-cost option. OpenAI documents GPTBot and OAI-SearchBot as independent settings, so disallowing GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search. It only works going forward: Anthropic describes a ClaudeBot block as excluding a site's future materials from its training datasets.

Blocking search crawlers takes you out of the answers. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links, and Perplexity recommends allowing PerplexityBot if you want to appear in its results.

Blocking user-triggered fetchers is the least predictable. Anthropic says disallowing Claude-User stops Claude retrieving your pages for user queries, but OpenAI and Perplexity say their fetchers act for a person and may not follow robots.txt.

Google works differently. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. AI Overviews and AI Mode are part of Search: Google's AI features guide names Googlebot, plus the nosnippet, data-nosnippet, max-snippet and noindex controls, as the way to manage them, and points to Google-Extended only for training and grounding in "some of Google's other systems". So blocking Google-Extended won't remove you from AI Overviews, and the controls that do also limit your normal search snippets. Apple is similar: pages that disallow Applebot-Extended can still appear in Spotlight, Siri and Safari results.

Copy-paste robots.txt rules

The file goes at the root of every host you want covered, since each subdomain needs its own. Allow about a day for changes to register; OpenAI, Perplexity and Meta all mention delays of up to 24 hours. One trap: a crawler that finds a group naming it ignores the User-agent: * group entirely, so copy any path rules you rely on, such as Disallow: /admin/, into the named groups too. For the syntax itself, see the robots.txt guide.

Block training, allow AI search. The middle ground most sites want. Note that Google-Extended also covers grounding in Gemini Apps, and Meta-ExternalAgent also covers some product indexing, so both opt you out of slightly more than training.

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Stay visible in AI search answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block all declared AI bots. This also removes you from AI search citations, and the user-triggered fetchers may visit anyway.

# Block every declared AI crawler and fetcher
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Allow everything. With no AI-specific groups, every bot follows your wildcard rules. If you don't have a file yet, the Robots.txt Generator writes one.

# No AI-specific groups: every bot follows these rules
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt is a request, not enforcement

Nothing technically stops a bot ignoring robots.txt. In August 2025 Cloudflare reported that when Perplexity's declared crawlers were blocked, it fetched content with a generic Chrome-on-macOS user agent from IP addresses outside its published ranges, and Cloudflare removed Perplexity from its verified bots list. In the same tests, Cloudflare said ChatGPT-User read robots.txt and stopped when disallowed. Perplexity disputed the report, saying Cloudflare had misattributed traffic from BrowserBase, a third-party cloud browser service it uses only occasionally, and arguing that fetching a page for a user isn't crawling. If a bot must stay out, enforce it at the edge:

Where llms.txt fits

llms.txt is a proposal, published at llmstxt.org, for a Markdown file that gives AI agents a short, curated guide to your most useful pages. It is not a crawler control: it grants and denies nothing, so it can't replace robots.txt. Google's Gary Illyes said in July 2025 that Google doesn't support llms.txt and isn't planning to, as reported by Search Engine Land, so don't expect it to affect Google Search or AI Overviews. It's cheap to add for agents that do read it, and the llms.txt Generator builds one from a site description and the links you choose.

Check which AI crawlers your site blocks

CheckSEO's AI Crawler Access Checker fetches a domain's robots.txt and evaluates it for 16 user agents, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Applebot-Extended, Bytespider and Meta-ExternalAgent, with Googlebot and Bingbot shown for contrast. For each one it reports allowed, blocked or partly restricted, plus the rule that decided it, so you can confirm an edit did what you intended.