For the first time in roughly a decade, automated traffic has overtaken human traffic on the web. Security firms reported bots accounting for just over half of all web requests in 2024, and figures reported during 2026 put the share higher still, driven largely by AI crawlers and AI agents. Exact numbers vary by source and how "bot" is defined, but the direction is clear.
For a website owner, that raises three questions: which bots are good for you, which are costing you money, and how do you tell them apart?
The bots visiting your site
| Type | Examples | Good or bad? |
|---|---|---|
| Search engine crawlers | Googlebot, Bingbot | Good: needed to appear in search |
| AI search crawlers | OAI-SearchBot, PerplexityBot | Usually good: help you appear in AI answers |
| AI user-triggered fetchers | ChatGPT-User, and similar agents fetching a page for a user's question | Usually good |
| AI training crawlers | GPTBot, ClaudeBot, Google-Extended (a control token), CCBot | Your choice |
| AI agents | Browsing agents acting for shoppers | Increasingly important; treat like customers |
| Monitoring and SEO tools | Uptime monitors, Ahrefs, Semrush | Useful in moderation |
| Scrapers | Price and content scrapers | Often unwanted |
| Spam and attack bots | Form spam, credential stuffing, vulnerability scanners | Bad |
| Ad fraud bots | Fake clicks on your ads | Bad: costs money |
Why it matters
- Server costs and speed: heavy crawling can slow your site or raise hosting bills.
- Analytics accuracy: bots inflate sessions and distort conversion rates.
- Security: login attacks and vulnerability scans are constant.
- Ad spend: invalid clicks waste budget.
- Visibility: blocking the wrong bots removes you from search or AI answers.
Step 1: See who's visiting
- Server or CDN logs: user agents, IPs and request rates. Cloudflare's dashboard shows bot traffic and verified bots.
- Search Console Crawl Stats: how often Googlebot visits and how fast your server responds.
- GA4: look for suspicious patterns: huge spikes with near-zero engagement time, odd countries, one page hit repeatedly.
- Our AI crawler check: shows which AI crawlers your robots.txt allows or blocks.
Step 2: Decide your AI crawler policy
This is the big decision for 2026. A common, balanced approach:
| Bot type | Common choice | Why |
|---|---|---|
| Search engines | Allow | Essential |
| AI search crawlers and user fetchers | Allow | Needed to be cited in ChatGPT search, Perplexity and others |
| AI training crawlers | Your choice | Allowing may help models know your brand; blocking keeps content out of training |
| Unknown scrapers | Block or rate-limit | No benefit |
Note that Google-Extended controls whether Google uses your content for Gemini model training; it doesn't remove you from Google Search or AI Overviews. Blocking Googlebot itself would remove you from Search entirely.
For syntax and examples, see how to write a robots.txt file for every crawler, and test yours with our robots.txt tester.
Step 3: Stop the bad bots
Robots.txt is a request, not a lock. To stop bots that ignore it:
- Firewall rules at your CDN to block known bad user agents, countries you don't serve (carefully) and suspicious patterns.
- Rate limiting on login, search, checkout and contact forms.
- Bot management features (Cloudflare's free Bot Fight Mode or paid bot management).
- CAPTCHA alternatives on forms, such as invisible challenges or honeypot fields, while keeping forms accessible.
- Strong authentication (passkeys or multi-factor) so stolen passwords don't work. Read passkeys explained.
Our guide to Cloudflare for small business websites covers the settings.
Step 4: Clean your analytics
- Keep GA4's known bot filtering on (it's automatic).
- Exclude internal and monitoring traffic.
- Build segments for engaged sessions to judge content and campaigns.
- Compare conversions with real orders or leads in your own systems.
Step 5: Protect your ad spend
- Check Google Ads' invalid click reports.
- Exclude placements and audiences that send obviously fake traffic.
- Judge campaigns on real conversions, not clicks. See server-side tracking.
Get ready for AI agents
Not all automated traffic should be blocked. AI agents shopping or booking on behalf of real people are, in effect, customers. Make sure your site works for them: accessible pages, clear forms, structured data and fast responses. Read WebMCP explained and the AI agent-ready website checklist.
The short version
Bots now make up more than half of web traffic. Allow search engines and, usually, AI search crawlers; decide deliberately about AI training crawlers; block or rate-limit scrapers and attack bots with firewall rules rather than robots.txt alone; clean your analytics; and make your site work well for legitimate AI agents.
Want your bot settings reviewed? Our maintenance and support team can audit your logs, robots.txt and firewall rules.




