Indexing problem guide
Robots.txt is blocking the crawl
A robots.txt rule controls crawling, not indexing. If Googlebot is disallowed, Google may still know a URL exists from links or a sitemap, but it cannot fetch the page to read its content, canonical or noindex directive. That is why a blocked URL can linger in results with little or no snippet.
Start here
Run the Test your robots.txt before changing templates. It checks what a public crawler can see; Google Search Console remains authoritative for Google’s own indexing decision.
Start with the exact host and crawler
Robots files are scoped to a host. example.com/robots.txt does not govern www.example.com, a subdomain, or a different protocol endpoint that redirects elsewhere. Fetch the file on the same canonical host as the problem URL.
Then test the rule for the crawler that matters. Googlebot, Bingbot and generic * groups can produce different answers. The most specific matching group wins, and the longest matching rule wins inside that group. On an equal-length tie, Allow takes precedence over Disallow.
Rules that cause surprising blocks
Disallow: /in the wildcard group blocks the entire host unless a more specific allow rule wins.- A path rule can block assets that a page needs to render, even when the document URL itself is allowed. Check CSS, JavaScript and image paths when rendered content is missing.
- A rule written for
Googlebotdoes not automatically apply to every Google crawler. Keep special groups intentional and short. - A 404 robots.txt is treated as no robots file. A 5xx response is a different failure: crawlers may stop fetching the site until the file is healthy again.
The safe repair sequence
Remove only the rule that blocks the intended public path. Do not use robots.txt to hide URLs that are already indexed; use a visible noindex directive, allow the crawler to read it, and then wait for the URL to be removed. Do not put secrets in robots.txt either: the file is public and is often the first place an attacker looks for interesting paths.
After publishing, fetch the file from the public internet, test the exact path as Googlebot and confirm the page itself returns 200 with a usable canonical. Add the URL to your sitemap only after the access rule is correct. Submitting a blocked URL through IndexNow can announce it to participating engines, but it cannot override robots.txt.
If the rule changes frequently, add a deployment test that requests robots.txt and checks a small list of must-crawl paths. That catches a template regression before it becomes an indexing report.
Not sure this is your problem? Start from how Google indexing works and what to expect, which covers the whole process and links to every diagnosis from one place.