Paste your robots.txt, pick a crawler and a path, and see exactly which rule applies.
Paste your robots.txt and test exactly which paths each crawler can reach, so a single misplaced rule never hides your important pages from Google or AI assistants.
01
Rule parsing
Every User-agent, Allow and Disallow line read the way crawlers read them.
02
Path testing
Test any URL path against the rules for a specific crawler.
03
Crawler-specific rules
See how Googlebot, Bingbot and AI crawlers are each treated.
04
Common mistakes
Blanket blocks, wrong syntax and rules that don't do what you think.
- 1
Paste your robots.txt
Copy it from yourdomain.com/robots.txt.
- 2
Pick a crawler and path
Choose a bot and enter a path such as /blog/ or /checkout.
- 3
Check the verdict
See whether it's allowed or blocked, and which rule decided it.
Allowed where it should be
Public pages are crawlable; admin, cart and internal search are blocked.
Overlapping rules
Allow and Disallow rules conflict. The most specific rule wins, which can surprise you.
Site-wide block
'Disallow: /' for all agents stops search engines crawling your whole site.
robots.txt is tiny but powerful. Left-over rules from a staging site, a plugin or an old redesign regularly block important pages, or the entire site, from search engines. Testing the rules takes a minute and can save months of lost traffic.
About the robots.txt tester.
Not reliably. It stops crawling, but a blocked page can still be indexed if other sites link to it. Use a noindex tag to keep a page out of results.