robots.txt Generator
🤖 Control search engine crawlers by creating a robots.txt file that tells bots which pages to crawl and which to avoid.
Configuration
Select Search Engines
Crawl Settings
Time between requests. 0 = no delay
Optional: Help search engines find your sitemap
Access Control
One path per line. Leave empty to allow all
Override disallow rules for specific paths
Quick Templates:
Preview
# robots.txt
Your robots.txt will appear here...
📚 robots.txt Best Practices
Do's:
- ✓ Place robots.txt in root directory
- ✓ Use lowercase filename
- ✓ Test before deployment
- ✓ Include sitemap reference
Don'ts:
- ✗ Don't block CSS/JS files
- ✗ Don't use for security
- ✗ Don't list sensitive URLs
- ✗ Don't forget wildcards
Common Paths:
- • /wp-admin/ (WordPress)
- • /cart/ (E-commerce)
- • /search/ (Search results)
- • /*.pdf$ (PDF files)