For the first time in roughly a decade, automated traffic has overtaken human traffic on the web. Security firms reported bots accounting for just over half of all web requests in 2024, and figures reported during 2026 put the share higher still, driven largely by AI crawlers and AI agents. Exact numbers vary by source and how "bot" is defined, but the direction is clear.

For a website owner, that raises three questions: which bots are good for you, which are costing you money, and how do you tell them apart?

The bots visiting your site

TypeExamplesGood or bad?
Search engine crawlersGooglebot, BingbotGood: needed to appear in search
AI search crawlersOAI-SearchBot, PerplexityBotUsually good: help you appear in AI answers
AI user-triggered fetchersChatGPT-User, and similar agents fetching a page for a user's questionUsually good
AI training crawlersGPTBot, ClaudeBot, Google-Extended (a control token), CCBotYour choice
AI agentsBrowsing agents acting for shoppersIncreasingly important; treat like customers
Monitoring and SEO toolsUptime monitors, Ahrefs, SemrushUseful in moderation
ScrapersPrice and content scrapersOften unwanted
Spam and attack botsForm spam, credential stuffing, vulnerability scannersBad
Ad fraud botsFake clicks on your adsBad: costs money

Why it matters

  • Server costs and speed: heavy crawling can slow your site or raise hosting bills.
  • Analytics accuracy: bots inflate sessions and distort conversion rates.
  • Security: login attacks and vulnerability scans are constant.
  • Ad spend: invalid clicks waste budget.
  • Visibility: blocking the wrong bots removes you from search or AI answers.

Step 1: See who's visiting

  • Server or CDN logs: user agents, IPs and request rates. Cloudflare's dashboard shows bot traffic and verified bots.
  • Search Console Crawl Stats: how often Googlebot visits and how fast your server responds.
  • GA4: look for suspicious patterns: huge spikes with near-zero engagement time, odd countries, one page hit repeatedly.
  • Our AI crawler check: shows which AI crawlers your robots.txt allows or blocks.

Step 2: Decide your AI crawler policy

This is the big decision for 2026. A common, balanced approach:

Bot typeCommon choiceWhy
Search enginesAllowEssential
AI search crawlers and user fetchersAllowNeeded to be cited in ChatGPT search, Perplexity and others
AI training crawlersYour choiceAllowing may help models know your brand; blocking keeps content out of training
Unknown scrapersBlock or rate-limitNo benefit

Note that Google-Extended controls whether Google uses your content for Gemini model training; it doesn't remove you from Google Search or AI Overviews. Blocking Googlebot itself would remove you from Search entirely.

For syntax and examples, see how to write a robots.txt file for every crawler, and test yours with our robots.txt tester.

Step 3: Stop the bad bots

Robots.txt is a request, not a lock. To stop bots that ignore it:

  • Firewall rules at your CDN to block known bad user agents, countries you don't serve (carefully) and suspicious patterns.
  • Rate limiting on login, search, checkout and contact forms.
  • Bot management features (Cloudflare's free Bot Fight Mode or paid bot management).
  • CAPTCHA alternatives on forms, such as invisible challenges or honeypot fields, while keeping forms accessible.
  • Strong authentication (passkeys or multi-factor) so stolen passwords don't work. Read passkeys explained.

Our guide to Cloudflare for small business websites covers the settings.

Step 4: Clean your analytics

  • Keep GA4's known bot filtering on (it's automatic).
  • Exclude internal and monitoring traffic.
  • Build segments for engaged sessions to judge content and campaigns.
  • Compare conversions with real orders or leads in your own systems.

Step 5: Protect your ad spend

  • Check Google Ads' invalid click reports.
  • Exclude placements and audiences that send obviously fake traffic.
  • Judge campaigns on real conversions, not clicks. See server-side tracking.

Get ready for AI agents

Not all automated traffic should be blocked. AI agents shopping or booking on behalf of real people are, in effect, customers. Make sure your site works for them: accessible pages, clear forms, structured data and fast responses. Read WebMCP explained and the AI agent-ready website checklist.

The short version

Bots now make up more than half of web traffic. Allow search engines and, usually, AI search crawlers; decide deliberately about AI training crawlers; block or rate-limit scrapers and attack bots with firewall rules rather than robots.txt alone; clean your analytics; and make your site work well for legitimate AI agents.

Want your bot settings reviewed? Our maintenance and support team can audit your logs, robots.txt and firewall rules.