X-Robots-Tag: syntax, use cases and how to check the header

The X-Robots-Tag is an HTTP response header that gives search engines the same rules as the robots meta tag, such as noindex or nofollow, without touching the HTML. It's the only option for files with no <head>, like PDFs or images, and it never appears in the page source, so you check it with curl -I or DevTools.

What the header looks like

A response carrying it looks like this:

HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex, nofollow

You can put several directives in one header, separated by commas, or send the header more than once. The ones you'll use most:

• noindex: keep the URL out of search results
• nofollow: don't follow the links on the page
• none: shorthand for noindex, nofollow
• nosnippet: show no text snippet in results
• noimageindex: don't index the images on the page
• unavailable_after: 2026-12-31: stop showing the URL after that date

A directive can also target one crawler by naming it first, as in X-Robots-Tag: googlebot: noindex. Without a name, it applies to every crawler. When rules conflict, Google follows the more restrictive one.

When to use it instead of meta robots

The meta tag needs an HTML page you can edit. The header works on anything the server sends back, which makes it the right tool for PDFs, Word files, images, feeds and JSON endpoints. A common case is a PDF price list that ranks above the product page it belongs to.

It also helps when you can change server config but not templates, or when one rule should cover a whole directory.

What it can't do is stop crawling. Google has to fetch the URL to read the header, so if robots.txt disallows that URL, the noindex is never seen. Our guide on how to check robots.txt covers that side.

Setting it on Apache, nginx and a CDN

On Apache with mod_headers, in .htaccess or the vhost:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

On nginx:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

One nginx catch: add_header inside a location block replaces any add_header lines inherited from the server block. On a CDN or hosting platform the same header is set through response header rules, for example Cloudflare Transform Rules, a Netlify _headers file or headers in vercel.json.

How to check it

From the terminal:

curl -sI https://example.com/file.pdf | grep -i x-robots-tag
curl -sI -A "Googlebot" https://example.com/ | grep -i x-robots-tag

The second line sends a Googlebot user agent, which catches rules that only fire for crawlers. In Chrome DevTools, open the Network tab, reload, click the document and read the Response Headers. In Search Console, URL Inspection shows the header as the reason when it blocks indexing, under "Indexing allowed?".

The mistakes that deindex whole sites

The classic one is a header left over from staging. Someone adds Header set X-Robots-Tag "noindex" to keep the test site out of Google, and the config ships with the launch. Nothing in the HTML gives it away. The guide to catching a noindex in production goes through the usual routes.

Next comes a rule that matches too much: a FilesMatch pattern meant for PDFs that catches every URL, a header set at server level instead of inside one location, or a CDN rule applied to the whole zone rather than a path.

Then there are mixed signals. A page with <meta name="robots" content="index"> and an X-Robots-Tag: noindex header gets the noindex, because the restrictive rule wins. The noindex tag guide explains how Google handles the two together.

Alerts when a noindex header appears

Deltio's SEO scanner reads the response headers as well as the HTML on every URL it scans. It looks for noindex in the meta robots and googlebot tags and in the X-Robots-Tag header, then compares each scan with the previous one. When a page that was indexable starts returning noindex, you get an email or Slack alert. Scans run automatically, with manual ones on top (3 a day on Starter).

The alert covers noindex. For other directives, curl is still the check. Starter costs £24 a month, or £20 billed annually, and you can try it free for 14 days.