How to Check Your robots.txt (and the Line That Deindexes a Site)

The robots.txt file is a few plain lines of text that most people never look at, and one of those lines can stop search engines from crawling an entire site. It is easy to read once you know the syntax, easy to get wrong, and one wrong edit can affect every page at once. Here is how to find and read a robots.txt, the single line that does the most damage, how to check one properly, and why a change to it is one of the most dangerous silent failures in SEO.

Deltio watches your client sites for robots.txt changes and alerts you on Slack the moment one appears. From £20 a month.

What robots.txt does, and what it doesn't

The robots.txt file sits at the root of a domain and tells search engine crawlers which parts of the site they may and may not crawl. It is a crawl-control tool, and understanding that narrow scope is the key to not misusing it.

The most important distinction: robots.txt controls crawling, not indexing. Blocking a URL in robots.txt asks Google not to fetch it. It does not reliably remove that URL from search results, and it is not the same as a noindex. In fact the two can work against each other, because if you block a page in robots.txt, Google cannot crawl it, which means it cannot read a noindex on that page either. If your goal is to keep a page out of results, noindex is the right tool and the page must stay crawlable. We cover that side in the noindex tag, explained.

How to find and read a robots.txt

Every robots.txt lives in the same place: the root of the domain, at yoursite.com/robots.txt. Type that into a browser and you will see the file for any site on the web. Reading it comes down to four directives:

• User-agent: names which crawler the following rules apply to. User-agent: * means all crawlers.
• Disallow: lists a path the crawler should not fetch. Disallow: /admin/ blocks that folder.
• Allow: carves out an exception inside a disallowed path.
• Sitemap: points to the site's XML sitemap so engines can discover it.

A typical, healthy file looks something like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /
Sitemap: https://yoursite.com/sitemap.xml

That says: all crawlers may access the site, except the admin and cart folders, and here is the sitemap. Nothing alarming.

The one line that can take a whole site down

Here is the line to fear:

User-agent: *
Disallow: /

Disallow: / blocks the entire site from being crawled. Every page, all at once. This is exactly the setting a staging or development environment uses on purpose, so search engines ignore it. The disaster is when that staging robots.txt gets deployed to production, which happens more often than anyone likes to admit. The live site keeps working perfectly for visitors. Crawlers, meanwhile, are told to stay out of everything, and over time the site's presence in search decays.

Because robots.txt is a single file at the root, its blast radius is the whole domain. A stray character in one line can affect every page on the site, which makes it one of the highest-stakes files an agency never thinks to check.

How to check a robots.txt properly

• Fetch it directly. Visit yoursite.com/robots.txt and read it. Look specifically for any Disallow: / under User-agent: *, and confirm the Sitemap: line points to the right, live sitemap.
• Use Search Console. Google Search Console has a robots.txt report that shows the fetched file, when it was last read, and any parsing issues. It is the authoritative view of what Google actually sees.
• Test a specific URL. Search Console's tools let you check whether a given URL is blocked, which is useful when only part of a site is affected.
• Run a crawler. A crawler will report which URLs are blocked by robots.txt across the whole site, good for catching a rule that blocks more than it should.

All of these answer the question "what does the robots.txt say right now?" None of them tell you when it changes.

Why robots.txt changes are so dangerous, and so invisible

Two things make robots.txt uniquely risky. First, its reach: one file, whole site. Second, its invisibility: nothing about the live site looks different when robots.txt breaks. Pages load, forms work, everything is green. The only symptom is a gradual loss of crawling and, eventually, rankings, showing up weeks later in a traffic graph long after the edit.

And robots.txt gets edited for all sorts of reasons that have nothing to do with SEO: a platform migration, a new hosting setup, a security plugin, a developer testing something. Any of them can copy a Disallow: / into production or drop the Sitemap: line, and unless someone happens to open the file, no one knows.

For an agency managing sites you don't control, that is the worst kind of risk: high impact, no warning, and nobody is watching. You cannot open every client's robots.txt every morning.

How Deltio catches robots.txt changes

Deltio fetches and checks the robots.txt on your client sites on a daily cycle. When the file changes, whether a new Disallow appears, the sitemap reference is dropped, or the whole thing flips to block crawling, you get a plain-language alert on Slack naming the site, the same day. That turns the single most dangerous file on a site from something nobody monitors into something you are simply told about. It watches the related indexing signals the same way, noindex, canonicals, and the sitemap itself, because these fail together more often than alone. See how the alerts are scoped in SEO change alerts.

The short version

robots.txt lives at the root of a domain, controls crawling rather than indexing, and is read through four directives: User-agent, Disallow, Allow, and Sitemap. The line to watch for is Disallow: /, which blocks the entire site and most often arrives by accident from a staging deploy. Checking the file tells you what it says today. On sites you are responsible for, what you actually need is to know the day it changes, because with robots.txt, by the time the traffic shows it, weeks of crawling are already gone.

Add your first client site, connect Slack, and Deltio starts watching the same day. Start your 14-day trial and make sure the next bad robots.txt reaches you first.