Build a correct robots.txt from a platform template, with the rules that actually matter and none of
the ones that do nothing. Runs entirely in your browser.
Two things this file cannot do
It cannot remove a page from search results
Blocking stops the crawl, not the listing. A blocked URL that other sites link to can still appear, without a description. Removal needs a noindex, and the crawler has to be able to fetch the page to see it.
It cannot keep anything private
robots.txt is public and is the first place anyone looks for interesting paths. Listing /admin/ advertises it. Protect things with authentication; use this file for crawl efficiency.
Then check it live
Once the file is published, test it against real paths. The rule that wins is the longest matching
one, not the first, so a file that reads correctly top to bottom can still behave differently from
how it looks.
Not strictly. A site without one is crawled entirely, which for most small sites is exactly what you want. It becomes worth having when you have areas that waste crawl attention (search result pages, faceted navigation, endless parameter combinations) or when you want to declare your sitemap location in the conventional place.
Will blocking a page here remove it from Google?
No, and this is the most common misunderstanding about the file. robots.txt stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it, listed without a description because nothing was ever fetched. To remove a page you need a noindex directive, and the crawler must be allowed to fetch the page to see it.
Should I block AI crawlers?
It is a genuine choice rather than a best practice. Blocking GPTBot, ClaudeBot and the rest keeps your content out of training data and, in some cases, out of AI-generated answers that might have cited you. Google-Extended is a separate case worth knowing: it controls generative AI training only and has no effect on your search rankings, so blocking it does not cost you Google traffic.
Where does the file go?
At the root of each host: https://example.com/robots.txt. It cannot live anywhere else and it is not inherited: example.com and blog.example.com each need their own, and neither covers the other.
Why does the generated file always end with a Sitemap line?
Because it is the one thing in robots.txt that is purely additive. Every other line restricts something; the Sitemap directive tells a crawler that has never seen your site where to find your list of URLs, without it having to guess at conventional paths.
How do I check the file works once it is live?
Use the robots.txt tester, which fetches the live file and tests specific paths against it with correct precedence. That matters more than it sounds, because the rule that wins is the longest matching one rather than the first, so a file that reads correctly top to bottom can still behave differently from how it looks.