Robots.txt Checker: How to Test Your File Before It Blocks Google
A single misplaced slash in robots.txt can quietly deindex an entire site — and because nothing visibly breaks, it can go unnoticed for weeks. Here's exactly what to check.
Why one file causes this much damage
Robots.txt sits at your domain's root (yoursite.com/robots.txt) and it's the first thing well-behaved crawlers check before touching anything else on your site. It's plain text, it's usually only a few lines long, and it's often the single easiest file on a site to edit by accident — which is exactly why it causes so many real incidents. A staging-environment robots.txt (which typically blocks everything) accidentally deployed to production is one of the most common, and most damaging, mistakes in SEO.
How robots.txt actually works
The syntax is deceptively simple, which is part of the problem — it's easy to assume you understand it fully after reading two lines:
User-agent:which crawler the following rules apply to (*means all crawlers).Disallow:a path crawlers shouldn't request.Disallow: /blocks the entire site — this is the single most common catastrophic typo.Allow:carves out an exception within a disallowed path.Sitemap:points crawlers to your sitemap file directly.
What trips people up: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still show up in Google's index (usually with no description, just the URL) if other sites link to it — because Google never crawled the page to see a noindex tag, it just knows the URL exists. If you actually want a page out of Google's index, robots.txt is the wrong tool; you need a noindex meta tag, which requires the page to be crawlable in the first place.
Fetch your live robots.txt and flag dangerous rules automatically with CheckSEO's Robots.txt Checker — no signup.
Open the Robots.txt CheckerThe most dangerous robots.txt mistakes
Disallow: / left in from staging
Blocks literally everything. This is almost always accidental — a staging config that never got swapped out before launch. Fix: check this first, every time, especially right after a site migration or replatform.
Blocking CSS and JS files
An older SEO habit was to block /assets/ or /js/ to "save crawl budget." Modern Google needs to render your page like a browser does to evaluate content and Core Web Vitals — blocking these files can make Google see a broken, unstyled page. Fix: never disallow CSS/JS directories unless you have a very specific reason to.
Confusing "disallow" with "noindex"
Covered above — disallowing a page you actually want deindexed doesn't remove it from Google's index, it just stops Google from seeing why it shouldn't be there. Fix: use noindex for removal, Disallow only for crawl management.
Case-sensitivity mistakes
Paths in robots.txt are case-sensitive. Disallow: /Private/ will not block /private/. Fix: match the exact casing your URLs actually use.
No sitemap reference
Not an error exactly, but a free win left on the table — adding your sitemap URL here is the most reliable discovery path for any crawler that checks robots.txt first (which is most of them). Fix: add Sitemap: https://yoursite.com/sitemap.xml.
Wrong file location
Robots.txt only works at the domain root. A file at yoursite.com/pages/robots.txt is simply ignored — crawlers only check yoursite.com/robots.txt. Fix: confirm it's actually served at the root, not nested in a subfolder by a CMS misconfiguration.
How to check yours right now
CheckSEO's Robots.txt Checker fetches your live file, flags syntax errors and dangerous rules, and shows exactly which paths are blocked — before you find out the hard way via a drop in Search Console impressions.
Frequently asked questions
How do I check if my site has a robots.txt file?
Visit yoursite.com/robots.txt directly in a browser. If it doesn't exist, crawlers assume everything is allowed — which is fine for most small sites, but means you have no crawl-control at all if you ever need it.
What does "blocked by robots.txt" mean in Search Console?
It means Googlebot found a URL (usually via a link) but was told not to crawl it. The page may still appear in search results without a description if it has external links pointing to it.
How do I fix a "blocked by robots.txt" error?
Find the matching Disallow rule for that URL's path and either remove it (if the page should be crawlable) or confirm it's intentional (if you're deliberately keeping crawlers out of that section).
Is robots.txt still necessary in 2026?
Yes — for managing crawl budget on larger sites, keeping crawlers out of admin/internal areas, and pointing crawlers to your sitemap. Small sites can function without one, but it costs nothing to have a correct one in place.
Can I block AI crawlers like GPTBot in robots.txt?
Yes — most AI crawlers (GPTBot, CCBot, Google-Extended, and others) respect a specific User-agent block the same way search crawlers do, so you can allow search indexing while opting out of AI training crawls separately.
Check your own site with the Robots.txt Checker.
Open Robots.txt CheckerMore from the blog
What Is Keyword Density, and How Much Is Actually Too Much?
Keyword density isn't the ranking factor most people think it is. The real formula, healthy ranges by content type, and a free checker to test your own page.
How to Validate Your XML Sitemap (and Fix the Errors Google Actually Cares About)
Validate your XML sitemap in seconds. See the 7 most common sitemap errors that block Google from crawling your pages, and exactly how to fix each one — free, no signup.