Robots.txt Tester
Fetch any site's robots.txt, see how it parses, and test paths against it: with the rule that decided each answer shown, not just a verdict.
Precedence, which is where testers disagree
Most robots.txt confusion comes from a file that mixes Allow
and Disallow. The specification is
clear and widely misimplemented:
- Group selection. The most specific matching
User-agentwins. If a file has a group for Googlebot, Googlebot ignores the*group entirely, including any rules it might have wanted. - Rule selection. Within that group, the longest matching path wins, not the first one in the file.
- Ties. When an Allow and a Disallow match with equal length, Allow wins.
That second rule is the one that catches people: putting a Disallow
first does not make it authoritative. Every result here names the matched rule so you can see which
line actually decided it.
The status codes run backwards
| robots.txt returns | What crawlers do |
|---|---|
| 2xx | The rules apply as written. |
| 4xx | Treated as no robots.txt at all. everything is crawlable. A broken file removes your rules rather than tightening them. |
| 5xx | Treated as a reason to stop crawling the site. If robots.txt is generated by your application, an outage can suspend crawling everywhere. |
Frequently asked questions
What happens if robots.txt returns a 404?
Nothing is blocked. A 4xx is treated as "this site has no robots.txt", so crawlers apply no restrictions at all. This surprises people who expect an error to be restrictive. A broken robots.txt does not lock crawlers out, it removes whatever rules you thought you had.
What happens if robots.txt returns a 500?
This is the dangerous one. Google treats a server error on robots.txt as a reason to stop crawling the site, rather than risk breaking rules it cannot read. If your robots.txt is generated by application code, an application outage can therefore suspend crawling of your entire site. Serving it as a static file avoids the whole class of problem.
Why do two robots.txt testers disagree about the same URL?
Almost always precedence. The specification says the most specific user-agent group wins, then within that group the longest matching rule wins, and Allow beats Disallow on an equal-length tie. Naive checkers apply rules in file order or let Disallow always win, which gives different answers on any file that mixes Allow and Disallow. This tool shows the matched rule so you can check the reasoning rather than trust it.
Do wildcards work in robots.txt?
Yes, though they are not in the original standard. An asterisk matches any run of characters and a dollar sign anchors the end of the path, so Disallow: /*.pdf$ blocks PDFs but not a URL that merely contains .pdf in a query string. All major crawlers support both.
Does robots.txt keep a page out of search results?
No. It stops the page being fetched, which is a different thing. A blocked URL can still appear in results if other pages link to it, showing without a description because nothing was ever crawled. To keep a page out of the index you need a noindex directive, and the crawler has to be allowed to fetch the page to see it.
Does one robots.txt cover my subdomains?
No. Every host serves its own, so example.com and blog.example.com need separate files, and neither inherits from the other. The same applies to protocols in principle, though in practice everyone redirects http to https.