AI Crawler Access Checker

See which AI crawlers your site lets in, which it blocks, and whether a firewall is blocking some without robots.txt saying so.

18 AI crawlersrobots.txt rules per pathFirewall and CDN testTraining vs search vs userURL or paste

About this tool

AI companies use several different crawlers, and they do different jobs. Some collect content for model training, some build the index behind AI search answers, and some fetch a single page when a person asks about it. Blocking all of them keeps you out of AI answers, and allowing all of them lets your content be used for training.

Enter your site and a path you care about. The checker reads your robots.txt and applies the real matching rules to each crawler, so you see who is allowed and which line decides it. With the live test switched on, it also requests your page using the user agent of six AI crawlers and compares the result with a normal browser. That finds the common case where robots.txt allows a bot but a CDN or firewall rule quietly returns 403.

Why use this tool

Grouped by purpose

Training, AI search and user-request crawlers are shown separately, so your policy is easy to read.

Firewall test

Detects bots that are blocked at the server even though robots.txt allows them.

Path aware

Rules are tested against the path you enter, including allow and disallow exceptions.

How it works

  1. Enter your website and the path to test.
  2. Optionally switch on the live request test.
  3. Review your policy, the table of crawlers and any firewall findings.

Frequently asked questions

Will blocking AI crawlers affect my Google rankings?
Blocking Google-Extended does not affect Search. It controls use of your content for Gemini training and grounding. Blocking Googlebot would, but that is a different crawler.
Which crawlers should I allow to appear in AI answers?
Search and user-request crawlers, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. Many sites block only the training crawlers and keep these allowed.
Does robots.txt really stop AI crawlers?
It is a request that the major companies say they follow. It does not stop a crawler that ignores it. Firewall rules or authentication are the only real protection.
Why is the live test optional?
It sends several requests to your site, each of which counts toward your daily checks. The test uses only a user-agent string, so it cannot prove what the real crawler sees from its own network, but it does reveal user-agent based rules.
Does this tool send my data to a server?
Only when you use a URL. To read a live page we fetch it from our server, because browsers cannot read other sites directly. Nothing you paste in as text or HTML leaves your browser, and fetched pages are not stored.
Are there usage limits?
Checks that fetch a live page count toward a daily allowance: 20 per day on the free plan, 300 on Pro and 1000 on Business. The allowance resets at 00:00 UTC. Pasted text and HTML are never limited.

More free tools

User Reviews

–
No ratings yet
Write a review
Your rating *

Your email is never published. Reviews are moderated, and links aren't allowed.

What users are saying

No reviews yet

Be the first to share your experience with this tool — your feedback helps others.