Should you block AI crawlers from your site? The short answer
No — not across the board, and definitely not by accepting whatever default a CDN or security plugin sets for you without checking it first. The honest answer is that “AI crawlers” is not one thing to allow or block, it is at least three different categories of bot with completely different jobs, and treating them as one decision is how businesses accidentally block the exact traffic that gets them cited in ChatGPT, Perplexity, or Google’s AI Overviews while leaving the door open to the bots they actually wanted to keep out.
This question stopped being theoretical in 2026. Cloudflare, which sits in front of a large share of the web, changed its default posture this year: as of 15 September 2026, training and “agent” bots are blocked by default on ad-monetised sites behind Cloudflare, while bots that power live search answers stay allowed. If you have not checked your own settings since that change, there is a real chance your site’s visibility inside AI assistants shifted without anyone on your team doing anything.
The three categories, and why lumping them together is the mistake
Every AI company now runs more than one crawler, and the names are easy to confuse on purpose or by accident. We group them into three buckets, which lines up closely with how Cloudflare itself now classifies traffic:
| Category | What it does | Example bots | Blocking it costs you |
|---|---|---|---|
| Training crawlers | Scrape content in bulk to train or fine-tune a model, with no connection to a specific user query | GPTBot, Google-Extended, ClaudeBot (training use), CCBot | Nothing in the short term — your content was already indexed by the time these models ship, and future citations do not depend on this specific crawl |
| Search / answer crawlers | Fetch pages in real time, or near it, to ground a specific answer the model is about to give a user, and typically drive a citation or link | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Everything — blocking these is the single fastest way to disappear from AI Overviews, ChatGPT answers, and Perplexity results for your own brand terms |
| User-triggered agents | Fetch a page because a specific person asked an assistant to look at it right now (browsing, research, shopping) | ChatGPT-User, Claude-User, Gemini (agent mode) | A broken experience for the actual human on the other end, who is often already your customer or prospect |
Multi-purpose crawlers complicate this further. Googlebot, Bingbot, and Applebot mix search indexing with training uses, and under Cloudflare’s new framework they get treated by the most restrictive rule that applies to any of their functions — which means a site owner who blindly blocks “training bots” can end up throttling general search indexing too if they are not careful about which specific user-agents they target.
Why would anyone want to block any of this?
Three reasons come up in almost every client conversation we have had on this since the start of the year, and they are legitimate, not paranoid:
- Content gets reused with no credit and no traffic back. A training crawler can ingest years of your blog, product copy, and pricing pages, and none of it has to result in a visit, a mention, or a dollar back to you.
- Server load and bandwidth, at scale. For a site with thousands of pages, aggressive crawling from a dozen different AI companies’ bots is a real infrastructure cost, not just a philosophical one.
- There is now a monetisation option, not just a block/allow switch. Cloudflare’s “pay per crawl” infrastructure lets a site owner charge AI companies for training access instead of simply blocking them, which turns this from a binary decision into a pricing one for sites with enough traffic to make it worthwhile.
None of those three reasons apply to the search or answer bots in the middle row of that table. Blocking those to solve a training-data complaint is treating a headache with the wrong medicine, and it is the single most common mistake we see when a well-meaning developer or IT provider “locks down the bots” after reading one article about AI scraping.
How do you actually tell them apart in your robots.txt?
Every major AI company publishes its user-agent string, and the fix is almost always a few lines, not a plugin:
- Audit what is already in your robots.txt and your CDN/WAF bot rules. If you are on Cloudflare, check Security > Bots > AI Crawl Control (or the equivalent in your current dashboard) before assuming your robots.txt is the only control in play — the platform-level default can override or sit alongside it.
- Allow search and answer bots explicitly, by name, rather than relying on a wildcard rule to let them through. Explicit beats implicit here because new bots launch often enough that a stale wildcard rule silently stops matching.
- Decide on training bots deliberately, not by default. If you have no monetisation plan and no strong objection, leaving them allowed costs you little and keeps your content inside the models that increasingly summarise your category to buyers who never visit a search engine at all.
- Never block user-triggered agents. If a real person’s AI browser is trying to check your pricing page or your store hours on their behalf, blocking that bot does not protect you from anything — it just gives that specific customer a dead end at the exact moment they were close to converting.
- Re-check this quarterly. New bots launch, existing ones get renamed, and platform defaults like Cloudflare’s change on their own schedule, not yours.
What we are telling our own clients to do before year-end
For almost every small-to-mid-size business we work with, the recommendation is the same: leave search and answer crawlers open, leave user-agent crawlers open, and make a conscious (not default) decision on training crawlers based on how much your content is worth to you versus how much the extra visibility in future model answers is worth. If your llms.txt file already signals what you want crawled, extend that same thinking to robots.txt rather than treating the two as unrelated.
If you manage your own hosting and have never looked at this, run a crawlability audit this week rather than next quarter — the Cloudflare default change alone is reason enough, and it is a fifteen-minute check against a problem that can quietly cut your AI citations to zero. If you want to see what a correctly configured site looks like to one of these answer engines specifically, we broke that down in our piece on how Claude decides what to cite.
Does blocking training bots actually hurt your AI search visibility?
Not directly, and this is the nuance that gets lost in most of the “should you block AI crawlers” takes we have read this year. A model’s training data is baked in at the point it was built; blocking GPTBot or a Google-Extended-style training crawler today does not retroactively remove your site from a model that already trained on it, and it does not stop that model from mentioning your business from memory. What it does affect is whether future model versions have fresh knowledge of you, and whether your newest content — a product launch, an updated price, a new location — ever makes it into that training set at all.
Search and answer crawlers are a completely different story, because those run close to real time. A question asked in ChatGPT or Perplexity this afternoon can trigger a live fetch of your page this afternoon. Block that bot and the model cannot read your current pricing page even if it wanted to recommend you — it will either skip you in favour of a competitor it can still read, or worse, answer from stale training knowledge that may already be wrong about your business.
Bottom line
“Block AI crawlers” and “allow AI crawlers” are both the wrong question. The right one is: which of the three categories is this specific bot in, and does blocking it protect something I care about or just quietly remove me from the answers my next customer is reading. Get that distinction right once, write it into your robots.txt and your CDN’s bot settings, and you will not have to think about it again until the next bot shows up with a name you do not recognise.