Read the robots directives a server sends in its HTTP headers. This is the only place PDFs, images and other non-HTML files can carry noindex.
Bulk URL checkPer-bot effective rulesDirective validationServer rule snippetsURL or paste
About this tool
X-Robots-Tag is an HTTP response header that does the same job as the meta robots tag, but it works for any file type. It can also be set globally in server configuration, which makes it easy to apply by mistake. A stray header on a template or CDN rule can noindex an entire site.
Directives can be limited to one crawler, as in "googlebot: noindex". This tool reads every header line, applies the bot-specific ones where they belong and shows the effective result for Googlebot and Bingbot, so you see what each will actually do.
Why use this tool
Reads every header line
Multiple X-Robots-Tag headers are combined the same way search engines combine them.
Bot-specific rules
Handles prefixes like googlebot: and bingbot: and shows the outcome for each.
Copy-ready config
Apache and nginx snippets for adding noindex to file types.
How it works
Paste the URLs of pages or files to inspect.
Choose HEAD for speed or GET if your server treats them differently.
Review the effective directives and fix any flagged values.
Frequently asked questions
When should I use X-Robots-Tag instead of a meta tag?
For files that have no HTML head, such as PDFs, images, video and downloads, or when you want to apply a rule to a whole folder through server configuration.
What if the header and a meta tag disagree?
Google combines them and applies the most restrictive result. A noindex in either place is enough to keep the page out.
Can robots.txt and X-Robots-Tag work together?
Not for noindex. If robots.txt blocks the URL, Google never requests it and cannot see the header, so the page may stay indexed without a snippet.
Does this tool send my data to a server?
Only when you use a URL. To read a live page we fetch it from our server, because browsers cannot read other sites directly. Nothing you paste in as text or HTML leaves your browser, and fetched pages are not stored.
Are there usage limits?
Checks that fetch a live page count toward a daily allowance: 20 per day on the free plan, 300 on Pro and 1000 on Business. The allowance resets at 00:00 UTC. Pasted text and HTML are never limited.
More free tools
User Reviews
–
No ratings yet
Write a review
What users are saying
💬
No reviews yet
Be the first to share your experience with this tool — your feedback helps others.