Crawl budget

Also called: The number of daily indexed pages of a website

Crawl budget is a term that refers to the amount of web pages indexed daily by Google's crawlers, the search engine software bots. The crawling of these bots (also known as spiders) is of great importance in indexing the content of various web pages (and the website in general). To do this, Google crawls a number of pages on the website every day. This number is called "the crawl budget." The crawl budget can vary slightly on a daily basis. However, this variation decreases over a long period of time if Google (or another search engine such as Bing) has found all the pages properly and knows where to place these pages.

Influencing factors on a crawl budget

There are a number of influencing factors on a crawl budget. Especially the size of a website (the number of pages), the number of visitors on the platform and the number of links to that website play an important role in this. The speed of the website also plays an important role in this. A Google bot needs time to index a page if this time is reduced the bots have more opportunity to crawl the entire website. Another factor is a 404 page. The moment Google's bots land on such a dead-end page, it negatively impacts your crawl budget. Finally, duplicate content(identical content on multiple pages) is not conducive to a crawl budget. After all, this has the effect that indexing will take two (or more times) longer than it should. Through search console (a webmaster tool) you can easily find out the crawl budget for a page.