Indexing tools inside your AI assistant

Checks every signal that can keep a URL out of a search index, in one pass: HTTP status, robots.txt, the robots meta tag, the X-Robots-Tag response header, and the canonical. Returns a verdict plus the specific rule responsible for each blocker. Also flags the robots.txt-plus-noindex combination, where the crawler never fetches the page and therefore never sees the noindex, so an already-indexed URL can persist indefinitely.

Follows one URL through every redirect hop and returns the status code, timing, mechanism (HTTP Location, meta refresh or JavaScript), cookies set and security headers at each step. Detects loops and chains that end on a 4xx or 5xx. Useful for finding a 301 that points at a dead page, which a plain status check of the starting URL hides because it reports the redirect's status rather than the destination's.

Fetches a site's robots.txt, parses every group, and tests whether given paths are crawlable, applying RFC 9309 precedence: the most specific user-agent group wins, then the longest matching rule, with Allow beating Disallow on an equal-length tie. Returns the exact rule that decided each answer. Also reports the status-code handling, which runs opposite to intuition: a 4xx on robots.txt means no rules apply at all, while a 5xx means crawlers should stay away entirely.

Returns the title, meta description, canonical, robots directives, Open Graph and Twitter card tags, hreflang alternates, language, viewport and H1s: read from the raw HTML as delivered, before any JavaScript runs. Tags injected client-side will therefore be reported as missing, which is deliberate: that is also how social crawlers see the page, and why a link preview can be blank when the tags look correct in a browser inspector.

Returns every response header with a plain explanation, and lists which common security headers are absent. Highlights X-Robots-Tag, which carries the same directives as the robots meta tag but as a header: invisible in the page source, and therefore the hardest indexing block to find by hand. Deliberately produces no numeric security score, because any such number would be invented.

Locates and reads an XML sitemap the way a crawler would: robots.txt Sitemap: directives first, then the conventional paths. Follows sitemap index files and totals across them. Returns the URL count, lastmod values and any parse errors. Pass a bare domain to have it discovered, or a full sitemap URL to read that one directly.

Takes up to 25 URLs and returns, for each, the status of the URL you supplied, the status after following redirects, how many hops it took, and the final destination. Reporting the first and final status separately is the only way to see a 301 that lands on a 404. URLs the request budget could not reach are returned marked as unchecked rather than silently omitted.

Fetches https://{host}/{key}.txt the way a search engine does and reports whether it would pass validation: verified, not_found, mismatch, bad_status or unreachable. This matters because the shared IndexNow endpoint, Bing and Yandex all answer 202 Accepted for a submission whose key was never published, validate it asynchronously afterwards, and discard it silently on failure, so a 202 is not evidence a submission will be honoured. Naver, Seznam, Yep and Amazon validate during the request and return 403 instead.

Returns a spec-compliant IndexNow key and the filename to publish it under. The key is not a secret: the protocol requires it to be world-readable at your domain root, and it proves control of the host rather than granting access to anything.

Returns the search engines this server can submit to, and, the useful part, whether each validates your key during the request or afterwards. Engines that validate asynchronously return 202 for a broken key; engines that validate synchronously return 403. Google is absent because Google does not participate in IndexNow.

Notifies participating search engines that URLs have changed. The key file is fetched and verified BEFORE anything is sent, and nothing is submitted if it does not resolve. Reaches Bing, Yandex, Naver, Seznam, Yep and Amazon, not Google, which does not participate. This has a real external effect: it asks live search engines to crawl the URLs given, so only call it for URLs that genuinely changed and that the user controls.