Meta robots checker

A meta robots checker for both places noindex hides

When a page vanishes from Google, the cause is very often a single noindex directive telling search engines to stay away — and the reason it's so hard to find is that it can live in two completely different places. Most tools only look at the HTML. This meta robots checker looks at both: the <meta name="robots"> tag in the page's HTML, and the X-Robots-Tag in the HTTP response headers. It reports exactly which directives are set, where they came from, and whether the page is currently allowed into the index.

Check the noindex tag in the HTML

The familiar one is the meta tag: <meta name="robots" content="noindex">, or its Google-specific sibling <meta name="googlebot" content="noindex">. When you check a noindex directive here, we parse the actual directives rather than just searching for the word — so content="none" is correctly read as noindex plus nofollow, and content="index, follow" is read as fully allowed. If a noindex tag is present, you'll see it flagged along with the exact content value, so you can go straight to that line in your template and remove it.

The X-Robots-Tag lookup most tools miss

The dangerous one is the header, because it's invisible when you view a page's source. A line like X-Robots-Tag: noindex set in your server config or CDN will de-index a page just as effectively as the meta tag, but you'll never spot it in the HTML — only in the response headers. We read that header directly from the origin server, and we handle the bot-specific form too, so X-Robots-Tag: googlebot: noindex is correctly reported as targeting Googlebot specifically. This X-Robots-Tag lookup is the piece that solves the "I checked the HTML and there's no noindex, so why is my page gone?" mystery.

noindex isn't always a mistake

One honest point: a noindex is a real, high-impact finding, but it isn't automatically an error. Staging sites, thank-you pages, internal search results, and admin areas are all legitimately kept out of the index on purpose. So we tell you plainly that the directive is there and where it lives — but whether it's a problem depends on whether you meant that page to be in Google. If you did, this is your answer for why it isn't.