Robots.txt Generator
Creates a robots.txt file for controlling search engine crawler access.
Lilu Baba's Robots.txt Generator builds a standard robots.txt file with your disallow rules and sitemap reference, ready to upload to your site's root.
How to use this tool
- Enter the target user agent (usually * for all crawlers).
- List paths to disallow, one per line.
- Optionally add your sitemap URL — the robots.txt content updates instantly.
FAQ
Robots.txt Generator — frequently asked questions
At the root of your domain, e.g. https://example.com/robots.txt — it won't be recognized anywhere else.
No — it only requests that compliant crawlers not access those paths; use a noindex meta tag for stronger control over indexing.