Robots.txt Tester & Validator
Paste a robots.txt, pick a bot and a path, and see instantly whether it's allowed.
2 user-agent blocks parsed.
Frequently asked questions
Why would my robots.txt accidentally block a bot I want to allow?
The most common cause is an overly broad Disallow: / rule meant for one bot that ends up matching more user-agents than intended, or a wildcard rule placed in a way that gets overridden by crawler-parsing order.
Does this tester check the live version of my robots.txt, or only what I paste in?
Only what you paste in — it doesn't fetch your site's live file automatically, so make sure you're testing your actual current robots.txt rather than an old copy.
What's the difference between "disallowed" and "blocked from AI training"?
Robots.txt disallow rules are directives, not enforcement — well-behaved bots respect them, but they're not a technical barrier. A hard block requires server-level restrictions, not just a robots.txt entry.
Can I test multiple bots and paths at once?
Test one bot-and-path combination at a time for a clear yes/no answer — for auditing several combinations, run through them one at a time so you can isolate which rule caused which outcome.