Robots.txt Generator

Create a robots.txt file with allow and disallow rules, sitemap links, crawl-delay and an option to block AI training crawlers such as GPTBot and CCBot.

  • Free, no sign-up
  • Runs in your browser
  • Your data stays private

How to use the Robots.txt Generator

  1. List the paths to disallow, and any exceptions to allow.
  2. Add your sitemap URL and choose the options.
  3. Copy robots.txt and upload it to your site root.

Why use this Robots.txt Generator

  • Rules and exceptionsDisallow paths and allow exceptions for any user-agent.
  • AI crawler blockOne tick adds GPTBot, CCBot, ClaudeBot, Google-Extended and more.
  • Sitemap linesAdd one or more full sitemap URLs.
  • ChecksWarns about paths without a leading slash and about blocking the whole site.

Frequently asked questions

Where does robots.txt go?

At the root of your domain, for example https://example.com/robots.txt.

Does robots.txt hide pages from Google?

It stops crawling but not always indexing. Use a noindex tag to keep a page out of search.

Will blocking AI bots work?

Well-behaved crawlers follow robots.txt. It is a request, not a technical block.

Related tools

More SEO & Web tools

User Reviews

–
No ratings yet
Write a review
Your rating *

Your email is never published. Reviews are moderated, and links aren't allowed.

What users are saying

No reviews yet

Be the first to share your experience with this tool — your feedback helps others.