Get Alerted When robots.txt Changes

A robots.txt changed alert tells you when a site's crawl rules have been edited, so a stray Disallow: / gets reverted before it costs the site its crawl. robots.txt is a plain text file that can stop Googlebot requesting any URL on a domain, and nothing on the site looks broken when that happens.

A typical sequence goes like this. A deploy copies the staging config to production, Disallow: / goes live, and the site keeps loading normally for visitors and for uptime checks. Impressions in Search Console start sliding a week or two later, and only then does someone open the file. This guide is about hearing sooner. For reading and validating the file itself, start with how to check robots.txt.

Which edits deserve an alert

Plenty of robots.txt edits are harmless, like a plugin adding a rule for its own admin path. These are the ones worth acting on the same day.

Change	Effect	Urgency
Disallow: / under User-agent: *	Blocks every URL for every crawler	Treat as an incident
New Disallow on a content folder such as /blog/ or /products/	Takes a whole section out of the crawl	Same day
New User-agent: Googlebot group	Googlebot follows only that group and ignores the * rules	Same day, read what it contains
File starts returning a 5xx	Google pauses crawling, then relies on a cached copy	Same day
File deleted or returning 404	Google treats the site as unrestricted, and the Sitemap: line is gone	Check, rarely urgent
Sitemap: line removed or pointing to an old URL	New pages get discovered more slowly	Low
Rules added for GPTBot, ClaudeBot, PerplexityBot or Google-Extended	Blocks AI crawlers, with no effect on Google Search	Confirm it was intended

The Googlebot row catches people out in both directions. Someone adds a group with a single Disallow: /search for Googlebot, and because Googlebot now ignores the * group, the /cart/ and /checkout/ rules that used to apply to it no longer do. Copy the full rule set into the new group, or it will be looser than intended.

The AI crawler row is sometimes a policy decision and sometimes a default switched on by a security or SEO plugin that nobody signed off. Either way, the client should know.

A blocked page can still be indexed

robots.txt controls crawling, and noindex controls indexing. A URL blocked in robots.txt can still appear in results, usually without a description, because Google knows about it from links but was never allowed to fetch it. Search Console reports those as "Indexed, though blocked by robots.txt".

The trap is combining the two. Put noindex on a page and also block it in robots.txt, and Google never crawls the page, never reads the noindex, and the URL can stay indexed. To take a page out of results, leave it crawlable and serve noindex. The noindex tag guide covers that half.

What Search Console tells you, and when

The robots.txt report in Search Console (Settings > robots.txt) shows the versions Google fetched, when it fetched them, and whether the fetch succeeded or hit parsing problems. It's useful for diagnosis once you know something is wrong, but it doesn't email you when a line changes. The first automatic signal is usually a rise in "Blocked by robots.txt" in the Pages report, which only appears after Google has recrawled URLs under the new rules. Google also caches robots.txt for up to 24 hours, so even its own fetch can trail the edit by a day.

Building a basic check yourself

With a handful of sites and access to a server, a small script run hourly from cron covers the basics:

#!/bin/sh
URL="https://example.com/robots.txt"
DIR="/var/lib/robots-watch"
mkdir -p "$DIR" && touch "$DIR/last.txt"
STATUS=$(curl -s -o "$DIR/new.txt" -w "%{http_code}" "$URL")
if [ "$STATUS" != "200" ]; then
  echo "robots.txt returned $STATUS" | mail -s "robots.txt error" you@example.com
  exit 1
fi
if ! cmp -s "$DIR/new.txt" "$DIR/last.txt"; then
  diff "$DIR/last.txt" "$DIR/new.txt" | mail -s "robots.txt changed" you@example.com
fi
mv "$DIR/new.txt" "$DIR/last.txt"

It checks the status code as well as the content, which matters because a persistent 5xx is as serious as a bad rule. The weaknesses appear as the site list grows. You need one entry per site and working mail delivery, you learn nothing about noindex or sitemap changes from the same deploy, and nobody notices if the cron job stops running.

Using Deltio for the alert

Deltio fetches robots.txt on every site you add as part of its daily check, compares it with the version from the previous check, and sends an alert to Slack and email for that site when it has changed. The same pass checks noindex, canonicals, titles and the URLs in the sitemap, so a staging deploy that breaks several things at once arrives as one set of alerts. SEO change alerts lists everything that's covered.

Frequency is the honest limit. With a daily check, an edit made just after a check is reported the following day, where an hourly script like the one above would catch it sooner. Given Google's own caching of the file and the time a recrawl takes, a same-day or next-day alert still normally arrives before the damage shows up in Search Console.

A good first site is the one that worries you most, whether that's the busiest deploy schedule or the developers who tell you the least. All plans include a 14-day free trial.