Enter your domain and we'll read your robots.txt to show which AI crawlers — GPTBot, ClaudeBot, Google-Extended and more — are allowed, blocked or not mentioned.
Check which AI crawlers your robots.txt allows, blocks or doesn't mention, from OpenAI's GPTBot and Anthropic's ClaudeBot to Google-Extended and PerplexityBot.
01
Your robots.txt
We fetch the live robots.txt file from your domain.
02
Every major AI crawler
GPTBot, ClaudeBot, Google-Extended, PerplexityBot and other AI user agents.
03
Allowed, blocked or unmentioned
What your rules mean for each crawler, in plain English.
04
Who runs each bot
The company behind each crawler and what it's used for.
- 1
Enter your domain
Type your website address.
- 2
Review each crawler
See which AI bots can read your site and which can't.
- 3
Decide and update
Change your robots.txt to match how you want AI systems to use your content.
Matches your intent
The crawlers you want are allowed, and any you don't want are blocked deliberately.
Not mentioned
Unmentioned bots follow your general rules, which may not be what you intended.
Blocked by accident
A broad rule blocks AI search crawlers you actually want, keeping you out of AI answers.
AI assistants increasingly answer questions people used to search for. If your robots.txt blocks the crawlers behind them, perhaps through an old rule or a security plugin, your content can't inform their answers. Equally, you may choose to block training crawlers. Either way, it should be a decision, not an accident.
About the AI crawler check.
It depends on your goals. Blocking training crawlers keeps content out of model training, but blocking AI search crawlers can keep you out of AI answers. Many businesses allow search-related bots and decide case by case on training.