robots.txt Tester
Free robots.txt tester. Fetch a site's live robots.txt, see its rules and sitemaps, and test whether any path is allowed for Googlebot, GPTBot or any user agent.
About the robots.txt Tester
Rules are matched the way Google does it: the most specific group for the user agent wins, the longest matching path wins, and Allow beats Disallow on a tie. * and $ wildcards are supported.
Related tools
All 73 tools →DNS Lookup
A, AAAA, CNAME, MX, NS, TXT, CAA and SOA records for any domain.
SSL Checker
Certificate validity, expiry, issuer and SAN names.
llms.txt Generator
Generate llms.txt and a draft ai.txt so answer engines understand your site.
FAQ Schema Generator
Turn Q&A pairs into valid FAQPage JSON-LD and copy a ready <script> tag.
FAQ
Is the robots.txt Tester free?
Yes. The robots.txt Tester is completely free, with no account, trial or credit card. To keep it fast for everyone, each visitor gets a fair-use rate limit.
Do you store what I check?
No. Our server fetches the URL or domain you enter, sends the result back to your browser and doesn't keep it. We only fetch public addresses.
What should I do after using it?
Fix what it found, then launch your product on ShipX: a free launch gives you a permanent product page with a dofollow link back to your site. Launch on ShipX
Ready to launch?
Launch free on ShipX and get a permanent product page with a dofollow link to your site. No audience required.