Robots.txt Generator

Choose what search engines and AI bots may crawl, and get a ready robots.txt file to upload to your website root.

Loading tool…

How to use the Robots.txt Generator

  1. Choose a default (allow all or block all) and list any paths to block, one per line.
  2. Optionally set rules for specific robots, such as blocking AI crawlers, and add your sitemap URL.
  3. Copy or download robots.txt and upload it so it opens at yoursite.com/robots.txt.

Frequently asked questions

Where do I put robots.txt?

In the root folder of your domain, so it is reachable at https://yourdomain.com/robots.txt. It does not work in a subfolder.

Does robots.txt hide a page from Google?

No. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex meta tag to keep a page out of search.

How do I block AI crawlers?

Click “Block all AI crawlers” to add Disallow rules for GPTBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, PerplexityBot and Bytespider.

Why must paths start with a slash?

Robots rules match the URL path from the root, so every Disallow or Allow value must begin with “/”, for example /admin/.