SEO glossary: Crawl budget

What is the crawl budget?

If Google defines a specific crawl budget, the search engine sets its own limits. The maximum number of subpages to be crawled varies from website to website.

 

The question here is how high the reputation of a domain is. The higher this is, the greater the maximum crawl budget. However, it is not only the maximum number of pages to be crawled that is defined in the crawl budget. It also determines how often the subpages of a domain are crawled and how in-depth an individual crawl is carried out.

 

 

What factors influence the crawl budget?

As already explained, the trustworthiness of the domain, among other things, is of decisive importance for the size of the crawl budget. But there are also other important factors that play a role here. These include

 

• Low-quality content and spam pages: If Google discovers pages with low-quality content or even spam, the crawl budget is reduced. In some cases, crawling may even be stopped completely.

 

• Duplicate Content: To avoid indexing of too much DUPLICATE CONTENT Google takes appropriate measures. Consequently, it comes to the fact that the crawling is significantly reduced.

 

• Hard and soft errors: If pages display an error, they offer users no added value. For this reason, Google endeavors to ban such pages from the index. If there are too many hard or soft 404 error messages occur, the crawl budget can be reduced.

 

• Session IDs and face ted navigation: If pages work with session IDs, they often lead to duplicate content. Faceted navigation offers users the opportunity to subdivide products into further criteria, for example. For example, color and size can be adjusted. Google sees little added value here, so the crawl budget is reduced.

 

What significance does the crawl budget have for search engine optimization?

Website operators are advised to take a close look at their pages at regular intervals. This makes it possible to control bots and adjust the crawl budget accordingly.

 

For example, if a domain contains many subpages with 404 errors or low-quality content, crawl optimization should be carried out. If all optimizations have been carried out, the ranking can then improve.