Build a simple robots.txt file or test the rules you already have. Enter allow and disallow paths, an optional sitemap URL, and a crawler product token. The result explains whether the supplied path matches an allow or disallow rule.
How to Use Robots.txt Generator & Tester
- Enter the crawler product token and the path or URL to test.
- Add allow/disallow paths, or paste an existing robots.txt file to test that text.
- Generate or analyze the rules and review the matched rule.
- Test representative allowed and blocked paths before publishing the file.
Test crawl rules before changing robots.txt
The generator creates a wildcard user-agent group for the paths you enter. If you paste an existing robots.txt file, that text takes precedence for the test. You can check a full HTTP or HTTPS URL or a path beginning with a slash; the matcher evaluates its path and query against the relevant rules.
Example: allow a public section inside a blocked directory
A disallow rule for /private/ can be paired with an allow rule for /private/public/. Test both a private page and a public page to confirm the intended distinction. The tester considers applicable user-agent groups, wildcard patterns, end anchors, and longest-match precedence.
Keep crawling and access control separate
Robots rules guide cooperative crawlers; they do not password-protect a directory. They are also different from page-level indexing directives. Review the Robots Exclusion Protocol when preparing more complex rules.
Use the Robots.txt and Sitemap Finder to locate the file returned by a public site. Check page-level robots metadata with the On-Page SEO Checker before concluding that a page can or cannot be indexed.
Key Features
- Browser-based processing
- Example inputs
- Copy and download results
- No paid API or signup
Common Use Cases
- Website publishing
- Content review
- SEO maintenance
Privacy
Your supplied content is processed in your browser and is not uploaded by this tool.
Limitations
Robots.txt controls cooperative crawling; it does not secure private content or guarantee removal from an index.