403 Forbidden and Search Crawlers

The page loads perfectly for you and returns 403 to Google. This is one of the least visible ways to lose search traffic, because nothing on your side looks broken.

Quick answer

A 403 Forbidden means the server understood the request and refused it. When it happens only to crawlers, something is deciding based on who is asking — a firewall rule, rate limit, geo-block or user-agent filter. A sustained 403 will eventually get an indexed page dropped from Google. Reproduce it with curl and a crawler user agent from outside your network, then allow-list the verified crawler rather than disabling the rule.

403 versus 404 versus 401

These get used interchangeably in conversation and mean quite different things to a crawler.

Status	Means	Crawler reads it as
403	I know what you want and I refuse.	Not available to me. Drop it if this persists.
404	There is nothing at this address.	Gone. Drop it after repeated confirmation.
401	Authenticate and try again.	Behind a login. Not indexable.
429	Too many requests, slow down.	Back off and retry — not a signal to drop the page.

That last row matters if you are rate limiting deliberately. Returning 429 tells a crawler to come back later; returning 403 for the same situation tells it the page is off limits. The second one costs you the page.

Why it happens to crawlers and not to you

Your browser arrives with a residential IP address, a mainstream user agent, cookies from previous visits, a referrer, and the ability to run JavaScript challenges. A crawler arrives with a datacenter IP, an unusual user agent, no cookies, no referrer, and no interest in solving a challenge. Any rule keyed on those differences separates the two without anyone intending it.

WAF and bot-management rules

The most common cause by a distance. Aggressive bot modes challenge or block anything non-browser-like, and a crawler that cannot solve a JavaScript challenge sees a 403.

Rate limiting that returns the wrong status

A crawler requesting many pages quickly is exactly what a rate limiter is built to stop. Returning 403 instead of 429 turns a temporary throttle into a permanent-looking refusal.

Geo-blocking

Googlebot crawls predominantly from US IP addresses. Blocking regions you do not serve can block the crawler that decides whether you appear in search at all.

User-agent blocklists

Often inherited from a copied config or a security plugin default. Worth reading line by line — these lists accumulate entries nobody remembers adding.

Hotlink and directory protection

Rules that require a same-site referrer break for crawlers, which send none. This also catches CSS and images, so the page gets fetched but renders as unstyled nonsense.

Security plugins blocking file types

Rules blocking direct access to non-HTML files apply to crawlers too. On WordPress this is a frequent cause of an IndexNow key file that exists but cannot be read.

The robots.txt case, which is backwards from what you expect

A 403 on /robots.txt does not block your site. Google treats a 4xx on robots.txt as meaning no robots.txt exists — so it applies no restrictions at all and crawls everything you thought you had excluded.

A 5xx on robots.txt is the genuinely dangerous one. Google reads server errors there as a reason to stop crawling the site rather than risk violating rules it cannot read. If your robots.txt is generated by application code, that means an application outage can suspend crawling of your entire site — which is a good argument for serving it as a static file.

Reproducing it

You cannot debug this from a browser on your own network, because you are the case that works. Ask the way a crawler asks:

# As a crawler
curl -I -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/your-page

# As a browser, for comparison
curl -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://example.com/your-page

Different status codes from those two commands identify the problem immediately. Run them from a server or cloud shell rather than your own connection — if the rule is IP-based, your home address may already be trusted.

Then confirm against the real thing: URL Inspection in Search Console, using "Test live URL", shows you exactly what Googlebot receives right now. That is the authoritative answer, and it is the one to trust when your curl test disagrees with it.

403 in IndexNow specifically

If you arrived here because an IndexNow submission returned 403, the meaning is narrower and more useful: your key was rejected. Naver, Seznam, Yep and Amazon validate the key file during the request, so they answer immediately when it is missing, unreadable, or contains something other than the key you submitted.

The shared endpoint, Bing and Yandex accept exactly the same broken submission with a 202 Accepted, validate later, and discard it silently. So a 403 from the strict engines is not a worse outcome than a 202 from the lenient ones — it is the same failure, reported honestly.

Check why your key is rejected Does Google support IndexNow?

Frequently asked questions

What does 403 Forbidden actually mean?

The server understood your request, recognised the resource, and refused to serve it. That is different from a 404, which means the server has nothing at that address, and from a 401, which means you need to authenticate and are invited to try. A 403 is a refusal, not an invitation — the server is not asking you for credentials, it is declining.

Why does my page load in my browser but return 403 for Googlebot?

Because something is making a decision based on who is asking rather than what is being asked for. Firewall bot rules, rate limiting, geo-blocking, and user-agent filters all do this. Your browser presents a residential IP, a real browser user agent, cookies and a referrer; a crawler presents none of that. Test with curl and a Googlebot user agent from outside your network and you will usually reproduce it immediately.

What happens to a page that returns 403 to Google?

If the page was already indexed, a persistent 403 will eventually get it dropped — Google treats it like other 4xx responses and concludes the page is no longer available. A short-lived 403 is usually survivable because Google retries. The danger is the slow version: a bot rule tightened months ago, quietly removing pages one at a time, with nothing in your own monitoring to show for it.

What happens if robots.txt itself returns 403?

This one surprises people. Google treats a 4xx response for robots.txt as though no robots.txt exists, which means no restrictions and it crawls everything. So a 403 on robots.txt does not block your site — it removes your crawl rules. A 5xx is the opposite and far more dangerous: Google treats server errors on robots.txt as a signal to stop crawling the site.

Why is my IndexNow submission returning 403?

For IndexNow specifically, a 403 almost always means key validation failed rather than that you are blocked. Naver, Seznam, Yep and Amazon validate the key during the request, so they answer 403 when the key file is missing, unreadable, or does not contain the key you submitted. The shared endpoint, Bing and Yandex accept the same broken submission with a 202 and discard it later, which is why the same setup appears to work on some engines and fail on others.

How do I fix a 403 that only affects crawlers?

Find the layer making the decision — CDN or WAF rules first, then server config, then application middleware, then security plugins. Verify the crawler is genuine using reverse and forward DNS rather than trusting the user agent, then allow it explicitly. Resist the instinct to disable the whole rule: allow-list the verified crawler and leave the protection in place for everything else.