Should I block AI crawlers in robots.txt?

It depends on whether you want AI tools to cite you. If being recommended by ChatGPT or Perplexity matters to your business, allow their search crawlers. Many businesses block training crawlers and allow search crawlers.

Where does robots.txt go?

At the root of your domain, so it loads at yourdomain.ca/robots.txt. Crawlers won’t look for it anywhere else, and each subdomain needs its own.

Does robots.txt stop a page appearing in Google?

Not reliably. A blocked page can still be listed if other sites link to it. To keep a page out of results, allow crawling and add a noindex tag instead.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI documents GPTBot as the crawler for model training and OAI-SearchBot as the one that surfaces sites in ChatGPT search. You can allow one and block the other.

Let’s Talk

Robots.txt generator

This generator writes a robots.txt file for your site. Choose which AI crawlers may read it, block any private folders, add your sitemap, and copy or download the file. It takes a minute and avoids the mistakes that quietly hide a site.

Search engines

Allow crawlingBlock everything (staging only)

Folders to keep out of crawling

One path per line. Robots.txt is public, so don’t list secret URLs.

Sitemap URL

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /thank-you/

Sitemap: https://yourbusiness.ca/sitemap.xml

Upload this to the root of your domain so it loads at /robots.txt.

Want me to check what your site tells crawlers?

Send me your website and I’ll check your robots.txt, sitemap and indexing for anything hiding your pages. Free.

How to use it

Choose whether search engines may crawl the whole site. Only block everything on a staging site.
List any folders that should not be crawled, one per line.
Decide, crawler by crawler, which AI bots may read your site.
Add your sitemap URL, then download robots.txt and upload it to the root of your domain.

Why it matters

Robots.txt is a plain text file at the root of your site that tells crawlers what they may read. It is a request, not a lock. Reputable crawlers respect it, though some companies say fetches a user asks for directly may not. It doesn’t hide anything from people, so never use it to protect private information.

It matters more now because AI companies run several different crawlers. Some collect training data. Others fetch pages so that ChatGPT, Claude or Perplexity can answer a question and cite you. Blocking the training crawler and allowing the search crawler is a reasonable middle ground for many businesses.

The most common mistake I see is a site launched with the staging rule still in place: “Disallow: /” for every crawler. It quietly removes the whole site from search.

Good to know

Search vs training crawlers
OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for search answers. GPTBot, ClaudeBot and Google-Extended relate to training. Fetchers that act on a user’s request, like ChatGPT-User, may not follow robots.txt, according to OpenAI. Check each company’s documentation for current details.
Google-Extended is a token
Blocking Google-Extended doesn’t remove you from Google Search. It controls whether your content may be used to train and ground Google’s Gemini models.
Noindex is different
To keep a page out of search results, use a noindex meta tag and let it be crawled. A page blocked in robots.txt can still appear as a bare link.

FAQ