A robots.txt file tells well-behaved crawlers which parts of your site they're welcome to fetch. It isn't a security mechanism — it's a request that legitimate crawlers (Google, Bing, and most other search engines) honor, listed as one or more User-agent blocks, each followed by Disallow and optional Allow rules, plus an optional pointer to your Sitemap.
Robots.txt Generator
Build a robots.txt from crawl rules, without writing the syntax by hand.
robots.txt
User-agent: * Disallow:
What goes into a robots.txt
Frequently asked questions
Where do I actually put the file this generates?
Save it as robots.txt (exactly that name, no extension changes) at the root of your domain — it has to be reachable at yourdomain.com/robots.txt specifically, not in a subfolder, or crawlers won't find it.
What does the empty 'Disallow:' line mean if I leave every field blank?
A Disallow line with nothing after the colon is the standard way of saying "no restrictions" for that user-agent — functionally identical to having no robots.txt at all for that crawler, but explicit about it rather than ambiguous.
When would I add more than one User-agent block?
When you want different rules for different crawlers — for example, letting Googlebot index everything while blocking a specific scraper bot, or blocking an AI-training crawler without blocking search engines. Each block only applies to the user-agent named directly above it.
Does a robots.txt rule stop a page from appearing in Google at all?
Not reliably — Disallow only asks crawlers not to fetch the page's content; if other sites link to it, Google can still index the URL itself (usually with no description) unless you also add a noindex meta tag on the page, which requires crawlers to be able to fetch it in the first place.