What is crawling?
Crawling is the process where search engine bots visit web pages, download their content and follow their links to discover more pages across the web.
Want to know if your site gets crawling right?
I’ll check it, free, and send a short note on what’s working and what isn’t.
In plain English
Googlebot starts from pages it already knows, fetches them and adds any new links to its queue. Sitemaps give it extra starting points. For JavaScript-heavy pages, Google also renders them, running the scripts to see the final content, which can happen later than the initial fetch.
Crawling only means Google has seen a page. Whether it’s stored and shown in results is a separate step called indexing.
Why it matters for a small business
A page that isn’t crawled can’t rank. Pages buried deep with no links pointing to them, links that only work through scripts, or sections blocked in robots.txt can all go unseen. Small sites rarely hit crawl limits, but they often have discovery gaps.
An example
A Hamilton plumbing company’s service area pages are only reachable through a search box on the site. Googlebot doesn’t type into search boxes, so it never finds them. Adding a plain “Areas we serve” list with real links in the footer gets all twelve pages crawled.
Common mistakes
Where I see crawling go wrong most often.
Linking to pages only through forms, buttons or script actions that crawlers don’t follow.
Blocking the scripts and styles Google needs to render the page.
Assuming a page is crawled because it exists, without checking Search Console’s URL inspection.
Crawling, answered
It varies. Pages that change often and get linked a lot are visited more frequently. Small sites may see important pages recrawled every few days to weeks.
Yes. Use URL Inspection in Google Search Console and request indexing. It speeds things up but doesn’t guarantee inclusion.
Keep reading
- nick@tavro.ca
- 587-439-9994
- Calgary, Alberta