Indexing and crawling

noindex, robots.txt, X-Robots-Tag, XML sitemaps and status codes: what keeps pages in Google's index, what pushes them out, and how to check.

How to Fix Index Coverage Errors in Search Console (and Catch Them First)

How to fix Index Coverage errors in Google Search Console by cause, and how to catch the on-page changes that create them before Google reports them.

Get Alerted When robots.txt Changes

Set up a robots.txt changed alert: which edits are dangerous, why crawl blocking is not noindex, and how to be notified the moment a rule changes.

Pages Removed From Your Sitemap: What to Do Next

Pages removed from sitemap? Triage in 4 steps: check status codes, find the cause, judge the real SEO impact and fix it today, plus daily monitoring.

How to Monitor Sitemap Changes Automatically

How to monitor sitemap changes: daily snapshots, diffs of added and removed URLs, lastmod shifts, and why Search Console will not alert you.

XML Sitemap Best Practices (and What Silently Breaks Them)

XML sitemap best practices: list only canonical, indexable URLs, keep the file current, respect the limits and avoid the mistakes that quietly break it.

How to Check If a Page Is Indexed on Google

How to check if a page is indexed on Google: a site: search, URL Inspection and the Pages report, why pages are not indexed, and how to stay ahead of it.

The Noindex Tag, Explained: What It Does and When to Use It

A noindex tag keeps a page out of search results. What it does, meta tag vs HTTP header, how it differs from nofollow and robots.txt, and when to use it.

How to Check Your robots.txt (and the Line That Deindexes a Site)

How to find, read and check a robots.txt file, the one line that can block a whole site from crawling, and why robots.txt changes are a silent SEO risk.

How to Catch a Noindex Before It Kills Your Rankings

A stray noindex quietly drops pages from Google. How it happens, how to check for it, and how to get alerted the moment it appears.