About

About xrobotscheck

A free, open resolver for conflicting robots directives across all three sources.

Whether a URL is crawled and indexed is decided in three different places: robots.txt (crawling), the meta robots tag and the X-Robots-Tag HTTP header (both indexing). They routinely disagree, and the rules for who wins are subtle — which is why “my page is noindex but still indexed” is one of the most common SEO confusions.

xrobotscheck reads the three places that decide whether a URL is crawled and indexed — robots.txt, the meta robots tag and the X-Robots-Tag header — for the crawler you choose, and resolves them with Google’s real precedence rules into one decision table. It names the traps (most notably noindex behind a Disallow), explains which source wins for each directive, and gives you a shareable result page. Open methodology, no black-box score.

The most valuable thing it catches is the noindex-behind-Disallow trap: because the tool fetches the page directly, it can see a noindex that Googlebot would never see (because robots.txt blocks the crawl) — and tell you why the page is still in the index. It runs on Cloudflare, fetches only the public URL and its robots.txt, and keeps no logs.

Related: aicrawlcheck (AI-crawler access) and soft404scan (soft 404s). Check a URL →