Check where a public website exposes its robots.txt and sitemap files. The finder inspects the origin-level robots location, reads sitemap declarations, and checks a bounded set of declared or common same-origin sitemap addresses.
How to Use Robots.txt & Sitemap Finder
- Enter a public website URL with the intended protocol and hostname.
- Run discovery and review the robots.txt response.
- Inspect declared, common, and cross-origin sitemap findings.
- Validate the sitemap XML and use Search Console separately for submission and indexing evidence.
Start with public crawl-discovery files
Enter a public website URL; the finder uses its origin for the robots.txt location. It then checks up to five same-origin sitemap candidates, including common sitemap.xml and sitemap_index.xml locations. Cross-origin declarations are reported for manual review rather than fetched as part of this discovery run.
Example: confirm a sitemap after publishing
After adding a sitemap declaration to robots.txt, run the finder to confirm the declared address and response status. A detected sitemap root is useful discovery evidence, but it does not establish that every entry is valid or that a search engine has read the file.
Separate discovery from validation and indexing
A file can be discoverable while containing malformed XML or unsuitable page URLs. A 404, blocked response, or request failure also requires context. The tool reports what this server received and does not inspect Search Console submission history or indexing status.
Use the XML Sitemap Validator to review downloaded XML. Use the Robots.txt Generator and Tester to inspect supplied rules, and the Redirect Checker for an individual file URL.
Key Features
- Bounded public website checks
- Example inputs
- Copy and download results
- No paid API or signup
Common Use Cases
- Website publishing
- Content review
- SEO maintenance
Privacy
The entered public URL is sent to our server for a bounded website check. Page content is processed for this request and is not saved by this tool.
Limitations
Checks up to five same-origin declared or common sitemap locations; it does not validate all sitemap URLs.