AI Crawler Access Checker

Enter a domain to see whether its robots.txt lets in or blocks GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and ten other AI crawlers — with the exact rule that decides each one. Free, instant, no signup.

What this checks & how to read it

It fetches /robots.txt from the root of the domain, parses it into rule groups, and then evaluates access to / for sixteen named user-agents: OpenAI's GPTBot, ChatGPT-User and OAI-SearchBot; Anthropic's ClaudeBot, Claude-Web and anthropic-ai; Google-Extended; Common Crawl's CCBot; PerplexityBot; Applebot-Extended; ByteDance's Bytespider; Amazonbot; Meta's meta-externalagent; Cohere's cohere-ai; and, for contrast, Googlebot and Bingbot. Each result shows the verdict, which User-agent group produced it, and the specific Allow or Disallow line that decided it.

How to change what it finds

Add a group per crawler you want to block, using the exact names above, for example User-agent: GPTBot followed by Disallow: /. Bots you do not name fall through to your User-agent: * group, so a site that blocks nothing by name is only as restrictive as its wildcard rules. Keep in mind the distinction between training crawlers and answer crawlers: blocking OAI-SearchBot or PerplexityBot also removes you from the citations those products show users. If you have nothing to block, the Robots.txt Generator can write a starting file for you.

Frequently asked questions

Does blocking GPTBot or ClaudeBot hurt my Google rankings?

No. Googlebot is a separate user-agent and is shown in the results purely for contrast. The one Google token to be careful with is Google-Extended: blocking it only opts your content out of Gemini training and does not affect Search or AI Overviews, which are driven by Googlebot. Blocking Googlebot itself, of course, removes you from Google.

Why are there three OpenAI crawlers, and do I need to block all of them?

They do different jobs. GPTBot collects pages for training future models. OAI-SearchBot indexes pages so they can be cited in ChatGPT search results with a link back to you. ChatGPT-User fetches a page live when a user asks about it. Many publishers block GPTBot but allow OAI-SearchBot and ChatGPT-User, because those two send visitors rather than train on content. The results list each one separately so you can make that choice deliberately.

What does 'partially restricted' mean?

The crawler is allowed to crawl the site, but the rule group that applies to it contains Disallow lines for specific paths — /admin/, /search/, /wp-admin/ and the like. That is normal and usually intended. The tool tells you which group it fell into and the first disallowed path so you can confirm nothing important is caught by it.

How does the tool decide which rule group applies to a bot?

The same way the crawlers themselves do under RFC 9309: a group whose User-agent line matches the bot's name, case-insensitively and by prefix (so 'User-agent: Claude' matches ClaudeBot), wins over the wildcard '*' group, and the most specific match wins if several apply. Groups sharing the same name are merged. If no group names the bot, the '*' group applies; if there is none, or no robots.txt at all, everything is allowed.

Do AI companies actually honour robots.txt?

The companies listed here all publish their user-agent names and state that they respect robots.txt, and the major ones demonstrably do. It is still voluntary — robots.txt is a request, not a lock — so treat it as the standard, low-effort way to state your policy rather than as enforcement. Bytespider in particular has been reported ignoring it; a firewall rule on the user-agent or its IP ranges is the only hard block.

Related tools

Redirect Chain CheckerSitemap ValidatorRobots.txt GeneratorRobots.txt Checker

Command Palette

Search for a command to run...