Should I block AI crawlers from my website?
Short answerFor most small businesses, no, at least not the search crawlers. Blocking them can stop AI tools citing or recommending you. AI companies now often run separate bots for search and for model training, so you can allow the first and block the second if you prefer. Decide deliberately rather than by plugin default.
Not sure which bots your site blocks?
Send me your site and I’ll check your robots file and firewall for AI crawler blocks. Free, within two business days.
The longer answer
Not all AI crawlers do the same job
The main AI companies use different user agents for different purposes. OpenAI has GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages a user asks it to open. Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User in a similar way. Perplexity uses PerplexityBot for its index and Perplexity-User for live fetches.
Google is different again. Google-Extended is a control token that tells Google not to use your content for training future Gemini models or for grounding in some Gemini features. It doesn’t affect Google Search, and it doesn’t remove you from AI Overviews, which use regular Googlebot.
The trade-off for a small business
A local service business generally wants to be found and recommended. Its pages describe services, areas and how to get in touch, not proprietary research. Blocking search crawlers mostly costs visibility and protects little.
Training is a separate question, and reasonable people differ. Some owners don’t want their writing used to train models. If that’s you, block the training bots and leave the search bots alone. The commercial impact of blocking training crawlers is less clear-cut than blocking search ones.
How to set it up
Rules go in your robots.txt. A User-agent: line names the bot and Disallow: / blocks it. Reputable AI companies say their crawlers respect these rules, though robots.txt is a request, not an enforcement mechanism. Fetchers that act on a user’s request, like ChatGPT-User and Perplexity-User, may not follow it the same way.
Also check your host or CDN. Some security services now block AI crawlers by default at the firewall, which overrides whatever your robots file says. A robots.txt generator makes it easier to set per-bot rules without typos.
What I’d do
Nick Lovett, founder of Tavro
On the sites I build, I allow search crawlers from Google, Bing, OpenAI, Anthropic and Perplexity by default, because clients want to be found. If a client prefers to opt out of training, I add specific blocks for the training bots and Google-Extended, and leave search open.
I also check the hosting firewall settings, because I’ve seen sites block AI search bots there without anyone realising.
Follow-up questions
Not by itself. ChatGPT search uses OAI-SearchBot. Blocking that one is what keeps your pages out of its search results.
No. Google says it doesn’t affect Search ranking or inclusion.
The major companies say their crawlers do, though user-requested fetches can be treated differently. Less reputable scrapers may not, and robots.txt can’t technically stop them.
Keep reading
- nick@tavro.ca
- 587-439-9994
- Calgary, Alberta