Robots.txt Generator
Generate robots.txt files with user-agent rules, allow/disallow paths, sitemap URL, and crawl-delay. Includes preset templates for common configurations.
Preset Templates
Global Settings
User-Agent Rules
Frequently Asked Questions
What is a robots.txt file and why do I need one?▾
A robots.txt file tells search engine crawlers which pages on your site they can or cannot access. It is placed at the root of your domain and is one of the first files crawlers check before indexing your site.
How do I block AI bots from crawling my website?▾
Use the "Block AI Bots" preset to generate rules that disallow GPTBot, ChatGPT-User, CCBot, anthropic-ai, and Google-Extended from accessing your site while still allowing regular search engines.
Can I add a sitemap URL to my robots.txt file?▾
Yes. Enter your sitemap URL in the Sitemap field and it will be appended to the generated robots.txt. This helps search engines discover and crawl all pages on your site.
Is my data safe when using this robots.txt generator?▾
Yes. All generation happens entirely in your browser. No data is ever sent to a server.