Is this robots.txt tester free?

Yes. You can run 3 checks a day without creating an account, and the result appears straight away on the page. If you need to watch robots.txt on several sites continuously, Deltio does that as a paid service with a 14-day free trial.

Does the result match what Google does?

The tester follows Google's documented interpretation of RFC 9309: the most specific user-agent group, `*` and `$` wildcards, longest matching rule wins and Allow wins a tie. Other crawlers that follow the standard should reach the same verdict, but some smaller bots handle wildcards or conflicting rules differently.

Can I test a robots.txt before publishing it?

Not with this tool. It downloads the live file from the site you enter, so it always reflects what crawlers actually see. To test a draft, publish it on a host that is publicly reachable and run the check there.

How do I block GPTBot without blocking Google?

Add a separate group with `User-agent: GPTBot` followed by `Disallow: /`. Googlebot ignores that group and keeps following its own rules or the `*` group. OpenAI also documents other user agents, such as OAI-SearchBot, so decide which of them you want to block.

Why does Search Console say blocked when the tester says allowed?

Google may still be using a cached copy of robots.txt, which it generally keeps for up to 24 hours. Also check that you tested exactly the same URL: protocol, subdomain and letter case all matter, because each host has its own file and path matching is case sensitive.

← Free website tools

Free robots.txt tester

Enter a page URL, pick a crawler and see whether robots.txt allows or blocks it, plus the exact line that decides. Free, with no sign-up.

How the tester reaches its verdict

Paste the full URL of a page, choose a user agent and run the check. The tool downloads /robots.txt from the root of that host and applies the rules the way Google documents them, following RFC 9309. Each host has its own file: https://shop.example.com and https://example.com are tested separately.

The first step is choosing the group. A crawler obeys one group only, the one whose User-agent line matches its name most specifically. If the file contains a User-agent: Googlebot group, Googlebot follows that group and ignores everything under User-agent: *. This catches people out all the time: a new Disallow goes under *, and Google carries on crawling because its own group says nothing about it.

The second step is matching the path. Every Allow and Disallow in the group is compared with the path and query string of your URL. * matches any run of characters and $ marks the end of the URL. Matching is case sensitive, so Disallow: /Admin/ does not block /admin/. When several rules match, the longest one wins. If an Allow and a Disallow of equal length both match, Allow wins. If nothing matches, the URL is allowed.

Reading the result

The verdict says allowed or blocked for the user agent you picked. Underneath you get the deciding rule with its line number, the user-agent group that was applied and the whole file with numbered lines, so you can see the rule in context.

Take this file:

User-agent: *
Disallow: /blog/
Allow: /blog/guides/
Disallow: /*.pdf$

For /blog/guides/robots-basics/ two rules match, Disallow: /blog/ and Allow: /blog/guides/. The Allow is longer, so the page is allowed. For /blog/guides/checklist.pdf you might expect the PDF rule to win, but Allow: /blog/guides/ is still the longest match, so the PDF can be crawled as well. For /files/price-list.pdf only the PDF rule matches and the URL is blocked. Wildcard rules tend to lose to long folder paths, which is why it pays to test the actual URL instead of reasoning about the file.

When the file is missing or the server fails

If /robots.txt answers with a 404 or another 4xx status, crawlers treat the site as having no restrictions and everything is allowed. For plenty of small sites that is fine.

A 5xx error, or a server that does not answer at all, is a different story. Google treats it as a temporary block on the whole site and pauses crawling until it can fetch the file again. A robots.txt that times out behind a firewall or CDN rule can stop crawling of every page even though no line in it blocks anything. If the tester reports a server error, open the file in a browser and check your hosting or CDN logs first.

What robots.txt does not do

Blocking a URL stops crawling, not indexing. A blocked page can still show up in search results when other pages link to it, usually with no description. And since Google cannot fetch the page, it never sees a noindex placed on it. To keep a page out of the index, leave it crawlable and use noindex: our guide to the noindex tag explains how, and how to check if a page is indexed shows how to confirm the result in Search Console.

robots.txt is also a request, not access control. Googlebot, Bingbot and GPTBot respect it. Plenty of other bots do not, so never use it to hide private content.

Fixing a page that is blocked by mistake

Start from the deciding line. If it is a broad rule such as Disallow: / left over from a staging site, remove it or narrow it down. If one page inside a blocked folder needs to be crawled, add an Allow longer than the Disallow it has to beat. If only one crawler is affected, look for a group dedicated to that user agent: rules you added under * do not reach it, so they need copying into its group.

Run the tester again on the same URL and user agent once the file is live. Google generally caches robots.txt for up to 24 hours. For an urgent fix, the robots.txt report in Search Console shows the version Google last fetched and lets you request a recrawl. For a fuller manual routine, see how to check robots.txt.

Why one test is not enough

This check tells you what the file says right now. robots.txt changes for reasons that have little to do with SEO: a deploy that ships the staging file to production, a plugin update, a developer blocking a bot that was loading the server. Nobody tells the SEO, and the first sign is usually a drop in crawled pages weeks later.

Deltio monitors the robots.txt of every site you add. When the file appears, disappears or changes, you get an email or Slack alert straight away with a diff of the lines added and removed, so you see the rule that changed rather than a vague "file modified". Plans start at £24 a month, or £20 a month billed annually, and there is a 14-day free trial.

Frequently asked questions

Is this robots.txt tester free?: Yes. You can run 3 checks a day without creating an account, and the result appears straight away on the page. If you need to watch robots.txt on several sites continuously, Deltio does that as a paid service with a 14-day free trial.
Does the result match what Google does?: The tester follows Google's documented interpretation of RFC 9309: the most specific user-agent group, `*` and `$` wildcards, longest matching rule wins and Allow wins a tie. Other crawlers that follow the standard should reach the same verdict, but some smaller bots handle wildcards or conflicting rules differently.
Can I test a robots.txt before publishing it?: Not with this tool. It downloads the live file from the site you enter, so it always reflects what crawlers actually see. To test a draft, publish it on a host that is publicly reachable and run the check there.
How do I block GPTBot without blocking Google?: Add a separate group with `User-agent: GPTBot` followed by `Disallow: /`. Googlebot ignores that group and keeps following its own rules or the `*` group. OpenAI also documents other user agents, such as OAI-SearchBot, so decide which of them you want to block.
Why does Search Console say blocked when the tester says allowed?: Google may still be using a cached copy of robots.txt, which it generally keeps for up to 24 hours. Also check that you tested exactly the same URL: protocol, subdomain and letter case all matter, because each host has its own file and path matching is case sensitive.