The Noindex Tag, Explained: What It Does and When to Use It

A noindex tag is an instruction that tells search engines to keep a page out of their search results. It can sit in the HTML as a robots meta tag or be sent as an HTTP header, and it is often confused with nofollow and a robots.txt Disallow, which do different jobs.

Used on purpose, it's one of the most practical tools in technical SEO. Placed by accident on a page that earns traffic, it removes that page from Google without any error message or visible change on the page itself.

What noindex does, and what it doesn't

When Googlebot crawls a page and finds noindex, the page is dropped from the index. From then on it doesn't appear for any query, however good its content or backlinks are.

The limits matter just as much. noindex doesn't stop crawling; Googlebot has to fetch the page to read the instruction at all. It doesn't delete the page or change anything for visitors, and it doesn't directly affect other pages on the site.

There is one long-term side effect. Google has said that a page left on noindex for a long time is eventually crawled less and its links are treated as if they were nofollow. A long-running noindex on a hub or category page can therefore weaken the internal links to the pages beneath it.

The meta tag and the X-Robots-Tag header

The robots meta tag goes in the <head>:

<meta name="robots" content="noindex">

name="robots" applies to every crawler that honours it, while name="googlebot" targets Google only. Directives can be combined with a comma, as in content="noindex, follow", which keeps the page out of results but lets crawlers follow its links (subject to the long-term caveat above).

The same instruction can be sent as an HTTP response header:

X-Robots-Tag: noindex

The header exists because not every file has a <head>. It is the only way to noindex a PDF, an image or a plain text file. On Apache, a .htaccess rule like this applies it to every PDF on the site:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

On Nginx, add_header X-Robots-Tag "noindex"; inside a location block does the same.

The header is also where unwanted noindexes hide. It never appears in the page source, so a page can look clean in View source while the server tells Google to drop it. You'll only see it in the response headers, either in the Network tab of your browser's dev tools or with curl -sI against the URL.

Noindex vs nofollow vs Disallow

Instruction	Where it lives	What it controls	Effect
noindex	Meta tag or HTTP header	Indexing	Keeps the page out of search results
nofollow	Meta tag or link rel attribute	Links	Tells engines not to follow links or pass value through them
Disallow	robots.txt	Crawling	Asks engines not to fetch the URL at all

The expensive mix-up is between noindex and Disallow. A Disallow rule stops Google from fetching the URL, which means it can never see a noindex on that page. The URL can stay indexed, usually shown without a description, because Google still knows about it from links. To remove a page from results, leave it crawlable and use noindex. Only after it has dropped out should you add a Disallow, if you also want to stop it being crawled.

A noindex line inside robots.txt was never an official rule, and Google stopped honouring it entirely in September 2019.

If a page has to disappear from Google quickly, the Removals tool in Search Console hides it from results for about six months. That's temporary. The noindex, or deleting the page, is what makes the removal stick.

Pages that should carry noindex

• Thank-you and order confirmation pages
• Internal site search results, which can generate an almost unlimited number of thin URLs
• Staging and development environments, ideally also behind a password
• Login, account and cart pages
• Thin variants that must exist for users, such as print versions

For duplicates that have an obvious main version, a canonical tag is often the better choice, since it consolidates signals into the main URL instead of simply discarding the duplicate.

How a noindex ends up on the wrong page

Almost always through a change nobody connected to SEO. A staging configuration or database is copied to production and brings its noindex along. On WordPress, "Discourage search engines from indexing this site" under Settings > Reading stays ticked after launch. An SEO plugin update or settings reset switches a whole post type to noindex. A developer adds a header rule to hide a section under construction and forgets it at launch.

The page keeps loading normally, so nobody on the client side notices anything. Google recrawls, reads the directive and drops the URL, and the traffic graph shows it weeks later. Removing the tag takes minutes. The delay in spotting it is what does the damage. The practical side of finding one is covered in how to catch a noindex in production.

Checking and monitoring noindex

For a single URL, Search Console's URL Inspection tool is the most reliable check. It reports whether indexing is allowed and whether a block came from the robots meta tag or the X-Robots-Tag header. The Page indexing report lists every URL Google found under "Excluded by 'noindex' tag", and it's worth reading after any major release to confirm the list contains only pages you meant to exclude.

Both depend on someone deciding to look, and both reflect Google's last crawl rather than the page as it is today. Deltio takes the other route on client sites. It checks the pages in each sitemap on a daily cycle, reads noindex from both the meta tag and the header, and alerts on Slack and email when a page that was indexable on the previous check no longer is. Robots.txt changes, canonical changes and URLs leaving the sitemap are compared the same way. If you want alerts limited to those events, see SEO change alerts, or try Deltio free for 14 days.