crawler
Search engine automated programs, like tiny robots, roam the web. They visit your website, read content, follow links, and report everything back to Google, Bing, and other search engines. This is how your pages get indexed and can appear in search results. You can control what they see through settings like robots.txt and sitemaps .
A crawler is the program that precedes the index. It follows links, loads pages, and passes them on for analysis. What it doesn't reach can't be ranked. Therefore, clean internal linking and a well-maintained sitemap are not optional extras, but essential.
Each website receives a limited budget of visits. This isn't a problem for a small company website with 80 pages. However, it is for an online store with 20.000 addresses: if filter combinations generate thousands of nearly identical addresses, the crawler spends its time there and finds the 200 most important pages less frequently. Response times under 500 milliseconds increase the number of visits Google is willing to send.
Back to the glossary