URL Indexability Checker

Find out whether a search engine is allowed to index a URL, and if not, exactly which rule is stopping it. Free, no account.

The five ways a page gets kept out

Indexability is not one setting. It is the absence of any of these, and each is configured in a different place by a different person, which is why pages go missing without anyone changing anything obvious.

The status code

Anything that is not a success is not indexable. A 403 is the one worth watching for, because the page usually loads fine in your browser: that has its own page.

robots.txt

Blocks the crawl, not the index. A blocked URL can still be listed if other pages link to it: with no description, because nothing was ever fetched.

The robots meta tag

A noindex in the HTML. Visible if you look, and easy to ship by accident when a staging template reaches production.

The X-Robots-Tag header

The same directives, delivered as a response header. Invisible in the page source, which makes it the hardest of the five to find by hand.

The canonical tag

Not a block, but it hands indexing to a different URL. A template that hardcodes the homepage canonical will quietly deindex a whole section.

The combination that traps people

Blocking a URL in robots.txt and giving it a noindex tag looks like belt and braces. It is the one combination that reliably fails. The crawler obeys robots.txt, never fetches the page, and therefore never reads the noindex, so a URL already in the index stays there, sometimes for years. To remove a page, let it be crawled and let the noindex be seen. This tool calls that out when it finds it.

Frequently asked questions

What does "indexable" actually mean?
That nothing on your side is preventing a search engine from putting the URL in its index. It does not mean the page will be indexed. Indexing is a decision the engine makes about whether the content is worth storing, and it declines plenty of technically indexable pages. This tool tells you whether you have left the door open, not whether anyone will walk through it.

Why is a page blocked in robots.txt still showing in search results?
Because robots.txt blocks crawling, not indexing. If other pages link to a URL, an engine can list it without ever fetching it, which is why those results appear with no description. Worse, if you block a page in robots.txt that also carries a noindex tag, the crawler never fetches the page and never sees the tag, so the URL can stay indexed indefinitely. This tool flags that combination specifically.

What is X-Robots-Tag and why does it matter?
It is a response header that carries the same directives as the robots meta tag. It matters because it is invisible: nothing in the page source shows it, so a noindex delivered this way survives every check made by looking at the HTML. It is usually set at the server or CDN level, often years earlier by someone who has left.

The canonical points at a different URL. Is that a problem?
Only if it is unintentional, which it often is. A canonical tag tells search engines to index the other URL instead of this one, so the page you are checking will generally not appear in results in its own right. That is correct for a duplicate or a paginated variant, and a serious bug when a template hardcodes the homepage or a staging domain.

Does this check the same things Google checks?
It checks the signals that are visible from a single fetch: the status code, robots.txt, the robots meta tag, X-Robots-Tag and the canonical. Google also weighs content quality, duplication across your site, and whether it considers the page worth the crawl, none of which can be read from outside. Search Console's URL Inspection is authoritative for your own site; this is faster, needs no verification, and works on sites you do not own.

Which user-agent should I test as?
Googlebot by default, since that is what most people mean. Test as Bingbot if you care about IndexNow-participating engines, because robots.txt files quite often carry rules for one crawler that do not apply to the other. The tool shows which group in the file matched, so a surprising result is traceable to a specific line.