Robots.txt Generator
Choose what search engines and AI bots may crawl, and get a ready robots.txt file to upload to your website root.
Loading tool…
How to use the Robots.txt Generator
- Choose a default (allow all or block all) and list any paths to block, one per line.
- Optionally set rules for specific robots, such as blocking AI crawlers, and add your sitemap URL.
- Copy or download robots.txt and upload it so it opens at yoursite.com/robots.txt.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your domain, so it is reachable at https://yourdomain.com/robots.txt. It does not work in a subfolder.
Does robots.txt hide a page from Google?
No. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex meta tag to keep a page out of search.
How do I block AI crawlers?
Click “Block all AI crawlers” to add Disallow rules for GPTBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, PerplexityBot and Bytespider.
Why must paths start with a slash?
Robots rules match the URL path from the root, so every Disallow or Allow value must begin with “/”, for example /admin/.