Fetch a site's robots.txt or paste your own, then test URLs as Googlebot, Bingbot or AI crawlers. You get the deciding rule, its line number, and a matrix of every major bot.
Longest-match rulesWildcards and $Search and AI botsLine-by-line viewURL or paste
About this tool
Robots.txt rules are matched by specificity, not by order. For a given crawler the longest matching path wins, and when an Allow and a Disallow match with the same length the Allow wins. This tester applies the same logic Google documents, including * wildcards and the $ end marker, and merges groups that name the same crawler.
Blocked by robots.txt is not the same as noindex. A blocked URL can still appear in results with no description if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or X-Robots-Tag header instead.
Why use this tool
Shows the deciding rule
Every result names the exact Allow or Disallow line that won, so you know what to edit.
Search engines and AI bots
One path checked against Googlebot, Bingbot, GPTBot, ClaudeBot and more at once.
Annotated file
Your robots.txt with matched rules highlighted and lint warnings for typos and ignored directives.
How it works
Enter a site URL to fetch its robots.txt, or switch to Paste the file.
Pick a crawler and list the URLs or paths you want to test.
Press Test access and open the tabs to see every bot and the highlighted file.
Frequently asked questions
Which rule wins when Allow and Disallow both match?
The rule with the longest matching path wins. If an Allow and a Disallow are the same length, Allow wins. Their order in the file does not matter.
Does Googlebot obey Crawl-delay?
No. Googlebot ignores Crawl-delay. Bing and Yandex honor it, and Google Search Console has no crawl-rate setting for normal sites.
What happens if robots.txt returns a 404 or a 500?
A 404 means no restrictions, so everything may be crawled. A 5xx error is treated as disallow-all for a while, which can pause crawling of the whole site.
Will blocking a page in robots.txt remove it from Google?
No. Blocking stops crawling, not indexing. Use a noindex directive on a crawlable page to remove it from results.
Does this tool send my data to a server?
Only when you use a URL. To read a live page we fetch it from our server, because browsers cannot read other sites directly. Nothing you paste in as text or HTML leaves your browser, and fetched pages are not stored.
Are there usage limits?
Checks that fetch a live page count toward a daily allowance: 20 per day on the free plan, 300 on Pro and 1000 on Business. The allowance resets at 00:00 UTC. Pasted text and HTML are never limited.
More free tools
User Reviews
–
No ratings yet
Write a review
What users are saying
💬
No reviews yet
Be the first to share your experience with this tool — your feedback helps others.