What Is Googlebot?
The crawler that fetches your pages, what it actually does with them, and how to tell a real one from the many that pretend to be it.
Quick answer
Googlebot is the software Google uses to fetch pages from the web. It requests URLs, reads the response, follows links, and passes what it finds to Google's indexing systems. Crawling, indexing and ranking are three separate stages — Googlebot only does the first. A page can be crawled and never indexed, and that is the single most common source of confusion about how any of this works.
Crawling is not indexing, and indexing is not ranking
Almost every confused question about Googlebot dissolves once these three are separated. They happen in order, they can each fail independently, and Search Console reports on them separately.
1. Crawling — Googlebot's job
Fetch the URL. Get bytes back. Follow the links inside. That is the whole of it. Blocked by robots.txt, a 403, a timeout, or a server error.
2. Indexing — a separate decision
Google decides whether the fetched page is worth storing. Thin content, near-duplicates, a canonical pointing elsewhere, or a noindex tag all end the process here. "Crawled — currently not indexed" in Search Console is exactly this outcome.
3. Ranking — a different system entirely
Being in the index says nothing about position. An indexed page competing badly and a page that was never indexed look identical from the outside: no traffic.
Getting crawled faster only helps if crawling was your bottleneck. If Google is crawling your pages and declining to index them, no amount of additional crawling changes the outcome.
There is more than one Googlebot
"Googlebot" is often used to mean every crawler Google runs, but they are distinct agents with different purposes, and blocking the wrong one has different consequences.
| Crawler | What it is for |
|---|---|
| Googlebot Smartphone | The primary crawler. Google indexes the mobile version of your site by default, so this is the one whose view of your page becomes your index entry. |
| Googlebot Desktop | Still crawls, but secondary. If your mobile and desktop pages differ in content, the mobile one wins. |
| Googlebot Image / Video / News | Feed the vertical search surfaces. Blocking these removes you from image or news results without affecting web results. |
| Google-InspectionTool | Fires when you use the URL Inspection tool or the Rich Results test. Seeing it in your logs means someone was testing, not that Google was crawling. |
| GoogleOther | A general-purpose fetcher for internal use that is deliberately not tied to search indexing. |
| Google-Extended | Not a crawler at all — a robots.txt token that controls whether your content is used for Google's generative AI training. Blocking it does not affect search rankings. |
That last distinction matters: people block Google-Extended
expecting an AI opt-out and worry they have hurt their rankings. They have not. It is a separate control.
Most "Googlebot" traffic in your logs is fake
The user agent string is a plain text header that anyone can set to anything. Scrapers set it to Googlebot constantly, because plenty of sites wave through anything that claims to be Google. If you are making access decisions based on that string, you are making them based on an unverified claim.
Verifying properly takes two DNS lookups:
- Reverse lookup the IP. A genuine Googlebot resolves to a hostname under
googlebot.comorgoogle.com. - Forward lookup that hostname and confirm it resolves back to the IP that made the request. Without this second step the check is spoofable, because reverse DNS records can be set by whoever controls the IP block.
Google also publishes its crawler IP ranges as a JSON file, which is the practical option if you are doing this on every request rather than while investigating something.
How Googlebot finds your pages
Links
The original mechanism and still the strongest. A page with no internal links pointing at it is genuinely hard for Google to find, sitemap or not.
Your sitemap
A list of URLs you are asserting matter. A hint, not an instruction — inclusion does not oblige Google to crawl or index anything.
Search Console
"Request indexing" in the URL Inspection tool, one URL at a time. Useful for a genuinely new or changed page; useless as a bulk strategy.
Not IndexNow
Google evaluated the protocol and did not adopt it. An IndexNow submission has no effect on Googlebot at all.
What actually stops Googlebot
In rough order of how often each turns out to be the cause when a page is not being crawled:
A robots.txt rule you forgot about
Frequently a staging rule that shipped to production. Check the live file, not the one in your repo.
A firewall or bot rule returning 403
The page loads perfectly in your browser and is invisible to Google. This has its own page because it is common and hard to spot.
No internal links to the page
Orphan pages exist in your sitemap and nowhere else. Google notices that nothing on your own site considers the page worth linking to.
Server errors or timeouts
Sustained 5xx responses cause Google to back off crawling the whole site, not just the failing URL.
A noindex tag — which does not stop crawling
Worth stating precisely: noindex stops indexing, robots.txt stops crawling. Blocking a page in robots.txt so Google cannot see its noindex tag is a classic way to keep a page in the index.