Robots.txt Checker

Enter a domain to fetch its live robots.txt, see which crawlers are restricted from what, and catch rules that quietly block search engines. Free, instant, no signup.

What this checks & how to fix it

It fetches /robots.txt from the root of whatever domain you enter — the one location every crawler reads — and checks for the two things that matter most: whether a User-agent: * group disallows the entire site, and whether a Sitemap: directive is present. It then breaks the file into its rule groups so you can see at a glance which crawlers are being told what.

How to fix what it finds

The file is plain text at your site's root — edit it directly, or through your CMS's SEO settings (Yoast and Rank Math both have a robots.txt editor). If you need to write one from scratch, the Robots.txt Generator builds the syntax for you. Changes take effect the next time a crawler fetches the file, usually within a day.

Frequently asked questions

Is it bad if my site has no robots.txt at all?

No. A missing file means 'no restrictions' — every crawler is free to crawl everything, which is exactly what most small sites want. The only thing you lose is the Sitemap line, which is a convenient way to point crawlers at your XML sitemap. If you have nothing to block, a two-line file with just a User-agent: * and a Sitemap: directive is plenty.

Does 'Disallow' stop a page from being indexed?

No, and this is the most common misunderstanding about robots.txt. Disallow stops crawling — Google won't fetch the page — but if other sites link to it, Google can still index the URL and show it in results with no description. To actually keep a page out of the index, let it be crawled and put a noindex meta tag on it instead.

Why does this tool check a whole domain and not a single page?

There is only ever one robots.txt per host, always at the root: yoursite.com/robots.txt. Whatever URL you enter, this tool reads the root file for that domain, because that's the file every crawler reads regardless of which page it's about to visit.

Why is 'Disallow: /' flagged so severely?

Under a 'User-agent: *' group, 'Disallow: /' tells every search engine to stay off the entire site. It's the correct setting for a staging or development copy — and the single most common way a site accidentally vanishes from Google when that same file gets deployed to production.

Do AI crawlers respect robots.txt?

The major ones publish their user-agent names and say they honor it: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI training crawler, separate from Googlebot), and CCBot (Common Crawl). Blocking them is a per-user-agent group in the same file — the rule-group list above shows which agents your file currently addresses.

Related tools

Redirect Chain CheckerSitemap ValidatorRobots.txt GeneratorPage Speed & Size Check

Command Palette

Search for a command to run...