About xrobotscheck

Whether a URL is crawled and indexed is decided in three places: robots.txt (crawling), the meta robots tag and the X-Robots-Tag header (both indexing). They routinely disagree, and the precedence rules are subtle — which is why "my page is noindex but still indexed" is so common.

xrobotscheck reads all three for the crawler you choose and resolves them with Google's real rules into one decision table. Its key catch is the noindex-behind-Disallow trap: because it fetches the page directly, it sees a noindex that Googlebot never would (robots.txt blocks the crawl) and explains why the page is still indexed. Runs on Cloudflare, keeps no logs.