Enter any page URL and check the domain's robots.txt rules, whether that page is disallowed, and whether it appears in the site's sitemaps.
Try:
60 free checks per hour · No signup required · Reads the live page
Frequently asked questions
What does this checker do?
It reads the domain’s robots.txt, reports the disallow and allow rules, tells you whether the page you entered is disallowed, lists the sitemaps the site declares, and checks whether your URL appears in one of them.
What if my page is disallowed in robots.txt?
A disallowed page is not crawled by the bots that obey robots.txt, so it cannot rank. Fix the rule in robots.txt — or move the page — if the page should be indexed.
What does it mean if my URL is not in a sitemap?
It means search engines have to discover the page another way, usually by following links. Adding important pages to your sitemap is the fastest way to make sure they get found.