Let’s Talk

What are AI crawlers?

AI crawlers are bots run by AI companies that fetch web pages, either to train models, to build a search index, or to read a page when a user asks about it.

Want to know if your site gets AI crawlers right?

I’ll check how AI search sees your business today and what would help it cite you, free.

No newsletter. Just what you asked for, from me. Privacy

In plain English

It helps to sort them by job. OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for fetches a person triggers. Anthropic has ClaudeBot, Claude-SearchBot and Claude-User. Perplexity has PerplexityBot and Perplexity-User. Common Crawl’s CCBot builds a public dataset many models train on.

Google-Extended isn’t a separate bot. It’s a robots.txt token controlling whether Google may use your content for Gemini models, and it doesn’t affect Search or AI Overviews. The major crawlers say they respect robots.txt, though some companies note it may not apply to fetches a user triggers directly.

Why it matters for a small business

You can choose. A common middle ground is to block training crawlers but allow search crawlers, so your pages can still be found and cited. Many AI crawlers also don’t run JavaScript, so a site that renders in the browser can look empty to them.

An example

A Calgary law firm wants to stay out of model training but still appear in ChatGPT and Perplexity answers. Its robots.txt disallows GPTBot, ClaudeBot, CCBot and Google-Extended, and allows OAI-SearchBot, Claude-SearchBot and PerplexityBot. The robots.txt generator builds that file in a couple of minutes.

Common mistakes

Where I see AI crawlers go wrong most often.

  1. Blocking every bot with “AI” in its description, including the search crawlers that would cite you.

  2. Not realizing your host or CDN now blocks AI bots by default, so your robots.txt choices never take effect.

  3. Assuming a new robots.txt rule removes content that was already collected for training.

FAQ

AI crawlers, answered

  1. It depends on your goals. If you want to appear in AI answers, allow the search crawlers. Blocking training crawlers is a reasonable choice that won’t stop you being cited.

  2. No. AI Overviews use Google’s normal search crawler. Google-Extended only governs use of your content for Gemini models.

Keep reading

Related terms

More in AI search

Answers and tools

Work with me

Let’s begin

Let’s build something you’re proud to share.

Let’s Talk