Every well-behaved crawler checks one file before it touches anything else on your site: /robots.txt. It's a plain text file, it's older than Google itself, and it still quietly controls which parts of a website get crawled — including, now, whether AI models like ChatGPT and Claude are allowed to read your content at all. Get it wrong and the failure mode is brutal: a single misplaced Disallow: / can deindex an entire site overnight.

The basic syntax

A robots.txt file is organized into groups, each starting with a User-agent line and followed by one or more rules:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /cart/help

Sitemap: https://example.com/sitemap.xml

User-agent: * applies to every crawler that doesn't have its own specific group further down the file. Disallow blocks a path (and everything under it); Allow carves out an exception within a blocked path — useful when you want to block a whole directory except one page inside it, as in the /cart/help example above. Sitemap is a global directive (not tied to a specific user-agent group) that tells crawlers where to find your sitemap, and it can appear anywhere in the file.

How crawlers actually decide: longest match wins

This is the part almost everyone gets wrong. When a crawler checks whether it can access a path, it doesn't just take the first matching rule — it finds the longest matching pattern among all the rules in its group, and if there's a tie, Allow wins. So:

User-agent: *
Disallow: /blog/
Allow: /blog/best-practices

A crawler checking /blog/best-practices sees two matching rules — /blog/ (7 characters) and /blog/best-practices (20 characters) — and the longer one wins, so that specific page is allowed even though its parent directory is blocked. Get the pattern lengths backwards and you'll block pages you meant to allow, or expose pages you meant to hide, without any error or warning.

Blocking (or allowing) specific crawlers

Separate User-agent groups let you treat crawlers differently — the single most practical use case being AI crawlers, since 2026's crawler landscape now includes bots whose entire purpose is training large language models, not indexing you for search.

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Disallow: /admin/
Allow: /

This setup blocks OpenAI's training crawler entirely, keeps the default rule for everything else, and gives Googlebot an explicit (if redundant) group. The crawlers worth knowing by name in 2026: Googlebot and Bingbot for traditional search, GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (specifically Gemini and AI Overviews training, separate from regular Googlebot), PerplexityBot, and CCBot (Common Crawl, which feeds many other AI training sets indirectly). Each respects its own User-agent group independently.

Tip Blocking GPTBot and ClaudeBot stops those companies training on your content, but it also stops ChatGPT and Claude from citing you in a live answer when a user asks something your page would have answered well. Decide deliberately rather than defaulting to "block everything AI" — for most businesses that want AI-search visibility, allowing the search-oriented bots (OAI-SearchBot, ClaudeBot acting as a browser, PerplexityBot) while blocking pure training crawlers is the more useful middle ground.

Mistakes that actually break sites

  • ◆A stray Disallow: / under User-agent: *. This blocks the entire site from every crawler that doesn't have its own more specific group. It happens most often when a staging-site robots.txt (deliberately locked down) gets copied to production during a launch and nobody remembers to change it back.
  • ◆Blocking CSS and JS. Google renders pages like a browser before indexing them, so blocking your asset directories can make Google see a broken, unstyled page and judge it accordingly — a mistake that was common advice a decade ago and is actively harmful now.
  • ◆Case sensitivity. Paths in robots.txt are case-sensitive. Disallow: /Admin/ does not block /admin/. If your CMS generates mixed-case URLs anywhere, check this carefully.
  • ◆Assuming wildcards work everywhere. Google and Bing support * (any sequence) and $ (end of string) in paths, but not every crawler does. Don't rely on wildcard patterns for crawlers you haven't confirmed support them.
  • ◆Forgetting it's public. Anyone can view yoursite.com/robots.txt. Don't use Disallow paths to "hide" sensitive URLs — you're publishing a map of exactly where they are instead.

Testing before you ship

A syntax that looks right can still resolve wrong once longest-match-wins is applied across every rule in a group — which is exactly the kind of thing that's easy to get subtly wrong by eye and expensive to get wrong in production. Paste your robots.txt into our free robots.txt tester and check specific paths against specific crawlers before you publish; it applies the same matching logic real crawlers use, including the AI-crawler names above. If you want to see exactly how AI crawlers in particular will treat your whole domain, the AI crawler check runs through the full list in one pass.

Robots.txt is one piece of technical SEO hygiene — pair it with a proper XML sitemap so crawlers know what to look at, not just what to avoid, and see our technical SEO services if you'd like a full crawl-access audit done for you.