What is a robots.txt changed alert?

It is a notification that fires when the `robots.txt` file of a website is edited, added or removed. Because `robots.txt` controls which crawlers may access which parts of a site, a single changed line can block search engines from the entire domain. An alert tells you within minutes instead of weeks.

How do I get notified when robots.txt changes?

You need something that fetches the file on a schedule, compares it with the previous version and notifies you when the content differs. Manual checks and Search Console reports will not push a notification to you in time. A monitoring tool such as Deltio does the fetch, the comparison and the email or Slack alert automatically.

Does robots.txt remove a page from Google?

No. `robots.txt` blocks crawling, not indexing. A blocked URL can still appear in search results, typically without a description. To remove a page from the index you must let Google crawl it and serve a `noindex` directive, because a `noindex` on a page blocked by `robots.txt` is never read.

What happens if robots.txt returns a 500 error?

Google can treat persistent server errors on `robots.txt` as if the whole site were disallowed, and pause crawling until the file responds correctly again. That makes a 5xx on this one file as serious as a `Disallow: /` rule, which is why the status code is worth monitoring, not just the content.

Does Deltio alert me when robots.txt changes?

Yes. Deltio detects when `robots.txt` appears, disappears or is modified on any monitored site, shows a diff of the lines that were added and removed, and sends an immediate notification by email and Slack. You can toggle the alert on or off per event type.

How much does robots.txt monitoring cost with Deltio?

Deltio plans start at £24 a month, or £20 a month billed annually, and include robots.txt monitoring alongside sitemap, SEO, uptime, SSL and domain checks. There is a 14-day free trial.

← Back to the blog

Get Alerted When robots.txt Changes

A robots.txt changed alert tells you the moment your crawl rules are edited, before a stray Disallow: / costs you the whole site. robots.txt is a plain text file that can stop Googlebot crawling every page you own, and nothing looks broken when it happens. Deltio monitors robots.txt on every client site, detects when the file appears, disappears or changes, shows a diff of the added and removed lines, and notifies you by email and Slack straight away. From £20 a month, with a 14-day free trial.

Every SEO has a robots.txt story. A deploy pushes the staging config into production, Disallow: / goes live, Googlebot stops crawling. Nothing looks wrong: the site loads, the pages render, analytics keeps ticking. Two weeks later impressions slide, pages drop out of the SERP, and only then does someone open the file. This guide covers the alerting side. For reading and validating the file, start with how to check robots.txt.

The smallest file with the biggest blast radius

robots.txt sits at the root of the domain (https://example.com/robots.txt) and decides which crawlers may request which paths. Usually a dozen lines. Any one of them can be expensive.

Disallow: / blocks every path for every user agent. Staging environments ship with it as standard, which is exactly why it ends up in production.
Directory rules like Disallow: /blog/ remove a whole section from the crawl without touching a single page.
A missing file is not fatal (a 404 reads as "crawl everything"), but the Sitemap: line went missing with it.
A 5xx response is the worst case: on a server error, Google can treat the site as fully disallowed until the file is fetchable again.

Blocking the crawl is not removing from the index

robots.txt controls crawling. noindex controls indexing. Confusing the two causes most robots.txt damage.

A page blocked in robots.txt can still show in search results, usually with no description: Google knows the URL exists but was never allowed to fetch it.
Block a page and put a noindex on it, and Google never crawls it, so it never reads the noindex. The page stays indexed.
To take a page out of the index, leave it crawlable and serve noindex. To keep it out of the crawl, use robots.txt and accept it may still be listed.

Get it the wrong way round and you deindex what you wanted to keep. The noindex tag guide covers the indexing half.

The changes that deserve an alert

Disallow: / appears: total block. Treat it as a production incident.
A new Disallow: on an important directory: blog, products, categories, pagination. Someone tried to save crawl budget and went too far.
The file disappears, 404s or starts returning 5xx: your rules and your sitemap reference are gone, and a persistent 5xx is as dangerous as a bad rule.
The Sitemap: line is removed: quiet, but discovery of new URLs slows down.
Rules targeting Googlebot: a User-agent: Googlebot block is rarely an accident and almost always a problem.
AI crawler blocks appearing: GPTBot, ClaudeBot, PerplexityBot, Google-Extended. Sometimes policy, sometimes a plugin default nobody approved.

Why manual checks do not catch it

Opening /robots.txt in a browser works exactly once: on the day you remember. Across 20 client sites, a weekly check means finding out up to seven days late, and a Friday deploy has the whole weekend to do damage. Search Console has a robots.txt report with the fetched versions and their status: useful for diagnosis, useless as an alarm. It does not email you when a line changes, and coverage warnings are a lagging indicator. By the time "Blocked by robots.txt" shows up, the crawl has already stopped.

How Deltio does it

Deltio watches robots.txt on every site you add and detects three things: the file appearing, disappearing, or changing. When the content changes you get a diff of the added and removed lines, so you see the exact rule that was introduced without comparing two files by eye. The notification goes out immediately by email and to Slack, with a link straight to the dashboard, in your language and timezone.

It runs next to the rest of the monitoring on the same site: daily sitemap snapshots, SEO checks on title, meta description, H1, canonical, lang and hreflang, uptime checks every 10 minutes, SSL warnings at 30 days and 1 day, domain expiry. One dashboard, one set of SEO change alerts for 5 to 50 client sites.

Start with one site

Add the site that worries you most: the busiest deploy schedule, or the client whose developers tell you nothing. Deltio fetches its robots.txt, keeps watching it, and tells you the next time a line moves. From £20 a month billed annually (£24 monthly), with a 14-day free trial.

Frequently asked questions

What is a robots.txt changed alert?: It is a notification that fires when the `robots.txt` file of a website is edited, added or removed. Because `robots.txt` controls which crawlers may access which parts of a site, a single changed line can block search engines from the entire domain. An alert tells you within minutes instead of weeks.
How do I get notified when robots.txt changes?: You need something that fetches the file on a schedule, compares it with the previous version and notifies you when the content differs. Manual checks and Search Console reports will not push a notification to you in time. A monitoring tool such as Deltio does the fetch, the comparison and the email or Slack alert automatically.
Does robots.txt remove a page from Google?: No. `robots.txt` blocks crawling, not indexing. A blocked URL can still appear in search results, typically without a description. To remove a page from the index you must let Google crawl it and serve a `noindex` directive, because a `noindex` on a page blocked by `robots.txt` is never read.
What happens if robots.txt returns a 500 error?: Google can treat persistent server errors on `robots.txt` as if the whole site were disallowed, and pause crawling until the file responds correctly again. That makes a 5xx on this one file as serious as a `Disallow: /` rule, which is why the status code is worth monitoring, not just the content.
Does Deltio alert me when robots.txt changes?: Yes. Deltio detects when `robots.txt` appears, disappears or is modified on any monitored site, shows a diff of the lines that were added and removed, and sends an immediate notification by email and Slack. You can toggle the alert on or off per event type.
How much does robots.txt monitoring cost with Deltio?: Deltio plans start at £24 a month, or £20 a month billed annually, and include robots.txt monitoring alongside sitemap, SEO, uptime, SSL and domain checks. There is a 14-day free trial.