What is robots.txt?

robots.txt is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they may or may not request.

Want to know if your site gets robots.txt right?

I’ll check it, free, and send a short note on what’s working and what isn’t.

No newsletter. Just what you asked for, from me. Privacy

In plain English

It always lives at yourdomain.ca/robots.txt. Each block names a crawler, or * for all of them, and lists paths to Disallow or Allow. It can also point to your XML sitemap.

It’s a request, not a lock. Well-behaved crawlers like Googlebot follow it; bad bots ignore it. And blocking a URL from crawling doesn’t remove it from results if other sites link to it.

Why it matters for a small business

It’s a small file with outsized power. One stray Disallow: / carried over from a development site can stop Google crawling everything. It’s also where you decide how to treat AI crawlers, which has become a real business decision.

An example

A Calgary architecture firm launches a new site and enquiries dry up over the next month. The robots.txt still reads Disallow: / from the staging server. Replacing it with a file that allows everything and lists the sitemap, made with a robots.txt generator, gets crawling back on track.

Common mistakes

Where I see robots.txt go wrong most often.

  1. Shipping a staging robots.txt that blocks the whole site to production.

  2. Using robots.txt to hide a page from search results, when noindex is the right tool.

  3. Blocking CSS and JavaScript folders, which stops Google rendering the page properly.

FAQ

robots.txt, answered

  1. It isn’t required, but having one that allows crawling and points to your sitemap is a tidy default and avoids error logs.

  2. No. The file is public and only polite bots respect it. Anything private should sit behind a login.

Keep reading

Let’s begin

Let’s build something you’re proud to share.

Let’s Talk