Test your robots.txt against every AI crawler and several paths at once, and copy ready rules for the policy you want.
18 AI crawlersMulti-path matrixMatching rule shownStarter rulesURL or paste
About this tool
When you write rules for AI crawlers, the details decide the outcome: a crawler follows its own group if there is one and otherwise the wildcard group, the longest matching rule wins, and an Allow beats a Disallow of the same length. It is easy to believe a bot is blocked when it is not, or the reverse.
Paste your robots.txt, or fetch it from a site, and list the paths you care about. The tool applies the matching rules for each crawler and path and shows the result with the rule responsible. It also points out syntax problems, and offers three sets of starter rules: block training crawlers only, block all AI crawlers, or allow them all.
Why use this tool
Crawler by path matrix
See at a glance which bot can reach which section.
Why it matched
The deciding rule is shown under each result.
Starter rules
Three ready-made policies that you can paste into your file.
How it works
Fetch robots.txt from your site or paste it.
List the paths to test.
Review the matrix, fix the rules and copy a starter policy if you need one.
Frequently asked questions
What happens to a crawler with no group of its own?
It follows the "User-agent: *" group. If there is no wildcard group either, everything is allowed.
Can I block one AI company but not another?
Yes. Each crawler has its own token. Put the tokens you want to block in a group with Disallow: / and leave the others out.
Do these rules affect my search ranking?
Rules for AI tokens do not affect Googlebot or Bingbot. Blocking Google-Extended does not change how Google Search ranks you.
Is crawl-delay supported?
Some crawlers honor it, but many do not. Googlebot ignores it. Rate limit at the server if you need a firm limit.
Does this tool send my data to a server?
Only when you use a URL. To read a live page we fetch it from our server, because browsers cannot read other sites directly. Nothing you paste in as text or HTML leaves your browser, and fetched pages are not stored.
Are there usage limits?
Checks that fetch a live page count toward a daily allowance: 20 per day on the free plan, 300 on Pro and 1000 on Business. The allowance resets at 00:00 UTC. Pasted text and HTML are never limited.
More free tools
User Reviews
–
No ratings yet
Write a review
What users are saying
💬
No reviews yet
Be the first to share your experience with this tool — your feedback helps others.