A robots.txt checker that reads your real crawl rules
Your robots.txt file is the first thing a search engine reads before crawling your site, and a single stray line in it can quietly remove pages from Google. This robots txt checker fetches the live file, parses it, and lays out exactly which crawlers are allowed or disallowed from which paths — grouped by user-agent, just as a search engine reads it. If you're trying to check robots txt code without squinting at a raw file, this shows it to you cleanly.
The one thing we always flag
Most Disallow rules are intentional — sites routinely block /admin, /cart, or internal search from being crawled, and that's good housekeeping, not a problem. So we don't cry wolf over individual rules; we simply show you what's disallowed and let you judge whether it's meant to be. The one case we do raise loudly is a whole-site block — a Disallow: / applied to all crawlers — because that removes an entire site from search, and it's almost always a mistake left over from a staging environment. If we see it, you'll know immediately.
XML sitemap validator
A sitemap tells search engines which URLs you want crawled, so a broken or missing one slows discovery of your pages. This XML sitemap tester finds your sitemap — either from the Sitemap: line in your robots.txt or at the default /sitemap.xml — and validates it: is it well-formed XML, is it a URL set or a sitemap index, and how many URLs does it contain? If it's an index, we list the child sitemaps it points to. If your robots.txt names a sitemap that doesn't actually load, we'll tell you, since that mismatch is a common and easily-missed error.
Why crawlability is the first thing to check
When indexation drops, the fastest diagnosis is almost always the crawl layer: a robots.txt that changed, or a sitemap that stopped resolving. Rather than log into a heavy platform, you can validate both here in seconds and rule them in or out. We report what the files actually contain — the crawl rules and the sitemap's structure — factually, without inventing hazards or scoring your SEO. It's the honest, fast check to run before you go looking deeper.
noindex.