X-Robots-Tag Checker

Read the robots directives a server sends in its HTTP headers. This is the only place PDFs, images and other non-HTML files can carry noindex.

Bulk URL checkPer-bot effective rulesDirective validationServer rule snippetsURL or paste

About this tool

X-Robots-Tag is an HTTP response header that does the same job as the meta robots tag, but it works for any file type. It can also be set globally in server configuration, which makes it easy to apply by mistake. A stray header on a template or CDN rule can noindex an entire site.

Directives can be limited to one crawler, as in "googlebot: noindex". This tool reads every header line, applies the bot-specific ones where they belong and shows the effective result for Googlebot and Bingbot, so you see what each will actually do.

Why use this tool

Reads every header line

Multiple X-Robots-Tag headers are combined the same way search engines combine them.

Bot-specific rules

Handles prefixes like googlebot: and bingbot: and shows the outcome for each.

Copy-ready config

Apache and nginx snippets for adding noindex to file types.

How it works

  1. Paste the URLs of pages or files to inspect.
  2. Choose HEAD for speed or GET if your server treats them differently.
  3. Review the effective directives and fix any flagged values.

Frequently asked questions

When should I use X-Robots-Tag instead of a meta tag?
For files that have no HTML head, such as PDFs, images, video and downloads, or when you want to apply a rule to a whole folder through server configuration.
What if the header and a meta tag disagree?
Google combines them and applies the most restrictive result. A noindex in either place is enough to keep the page out.
Can robots.txt and X-Robots-Tag work together?
Not for noindex. If robots.txt blocks the URL, Google never requests it and cannot see the header, so the page may stay indexed without a snippet.
Does this tool send my data to a server?
Only when you use a URL. To read a live page we fetch it from our server, because browsers cannot read other sites directly. Nothing you paste in as text or HTML leaves your browser, and fetched pages are not stored.
Are there usage limits?
Checks that fetch a live page count toward a daily allowance: 20 per day on the free plan, 300 on Pro and 1000 on Business. The allowance resets at 00:00 UTC. Pasted text and HTML are never limited.

More free tools

User Reviews

–
No ratings yet
Write a review
Your rating *

Your email is never published. Reviews are moderated, and links aren't allowed.

What users are saying

No reviews yet

Be the first to share your experience with this tool — your feedback helps others.