In the Digital Marketing and SEO world in particular, the concept of ‘Crawl Budget’ is often overlooked because it is not easy to digest and seemingly complex to deal with. Understanding how search engines work includes topics that, at first glance, appear tough but are essential for the cooperation between our site and Googlebot. In this article, we will analyse in detail the notion of Crawl Budget, its impact on a site’s visibility in search engines, and all the main methods for optimisation and monitoring.
What is the Crawl Budget?
Imagine Googlebot (or any other search engine crawler) acting as a ‘spider robot’, in charge of exploring the vast labyrinth of ever-growing web pages. This spider operates 24 hours a day, seven days a week, but has limited available resources. The crawl budget represents the numerical value of these resources: the number of pages this spider ‘can’ and ‘will’ visit on a website in a given time period.
But why ‘can’ and ‘will’? The concept behind the ‘can’ is simple: Googlebot has, as mentioned, a limited amount of time to visit the contents of a website, so the faster the website is, free of empty, duplicated or self-generated pages, the easier its work will be.
For the ‘will’ the topic is wider. Google’s crawler spends an increasing amount of time and resources in the long term on websites that have a better structure, continuously updated content, greater authority, and in general greater relevance to web users. If a site is old, with little (or zero) content published annually, Googlebot will allocate fewer and fewer resources and time on that domain.
As we can imagine, the impact of crawl budget management has tremendous consequences (both positive and negative) on SEO performance: let’s analyse these in the next paragraph.
Why is the crawl budget vital in SEO?
The crawl budget becomes crucial when we appreciate its deep impact on a site’s ranking in search engines. Indeed, the frequency of Googlebot ‘visiting’ the pages of our site directly affects the search engines’ perception of the site’s authority and relevance. In simple terms, the more often Googlebot visits a website, the more authority it will attribute to its domain and viceversa. And, as widely known, a domain with higher authority is more likely to obtain higher positions in search engine results pages (SERPs).
In addition, the crawl budget is closely related to the concept of ‘link juice’. Each link shares a part of its ‘authority’ from one page of a website to another. This ‘link juice’ flows through the website, helping to determine which content is considered more important. If the crawl budget is well managed, ensuring that the most relevant pages are crawled and indexed regularly, it helps to effectively distribute this ‘link juice’ through the website.
In general, a good crawl budget management ensures that the most valuable pages are always in the Googlebot spotlight. This means that they will be crawled more frequently, contributing to their authority and higher ranking chances. On the other hand, less relevant or outdated pages should receive less attention, preventing them from taking valuable resources away from more valuable content.
In the next section, we will explore the main strategies to optimise the crawl budget and maximise the SEO performance of a website.
Optimising the Crawl Budget: Methods and Tools
Now that we have understood what the crawl budget is and why it is important, it is time to explore the strategies and tools that come to our aid to optimise this crucial resource and maximise our website’s performance. Here are some of the key techniques we can implement:
Improving website speed
As mentioned, the crawl budget represents the number of pages visited by Googlebot in a specific time frame: a particularly slow
website may also slow down the crawler’s job. On the other hand, a fast website not only improves the overall user experience, but also encourages the bot to visit more and more pages.
To monitor website speed, we can use free tools such as PageSpeed Insights or the Core Web Vitals section of Google Search Console.
Efficient site structure
A clear, logical and intuitive website structure simplifies navigation for both users and search engines. Everything should be organised hierarchically, with a clear map of categories, subcategories and product or service pages.
Furthermore, the density of internal links arranged in a reasonable and natural way within each page, distributes the crawl budget towards the sections of the site that have greater strategic relevance or are more up-to-date.
To check that the website structure is efficient, one method is to carry out a crawl through Screaming Frog and using the ‘Visualisations’ or the ‘Site structure’ section:
Absence of redirects and 404 pages
When a page is redirected (status code 3xx) or removed (status code 4xx) but is still linked to the site structure or in sitemap, Googlebot may still ‘visit’ it. This unnecessarily wastes the crawl budget because useful resources for crawling other or new pages are used to discover URLs that are no longer relevant for users and the search engine.
Whenever a page is redirected or in particular removed, it is necessary to unlink it from the site structure and remove it from the sitemap in order to prevent the bot from wasting time and resources to understand the content.
Here again Screaming Frog comes to our aid, which, in the Bulk Export section, allows us to export all internal links characterised by status code 3xx and 4xx and then proceed to remove them.
Sitemap optimisation
The XML sitemap should only contain URLs of pages that are relevant for SEO purposes in order to highlight to Googlebot which URLs to allocate Crawl Budget to. Furthermore, every time a new page is created, its URL should be added to the XML Sitemap as soon as possible. However, this is not enough: in addition to adding each new URL to the sitemap, it is important to link each new page to the website structure to avoid having orphan pages, which are generally crawled less than those in structure.
Both Crawlers (Screaming Frog, DeepCrawl, Oncrawl, etc.) and Google Search Console provide information on the ‘health’ of the sitemap and any errors.
Facet Navigation Management
Suppose we have an E-commerce with hundreds of Product Listing Pages (PLPs) with various kinds of filters (price, size, etc.): often selecting parameters in the filters is like landing on a new PLP with a new URL. This creates a very large number of similar pages, as many as the different combinations of filters.
Managing this kind of situation is one of the most difficult and important tasks in order to avoid wasted crawl budgets: Googlebot could scan these pages but then not find any new products or content.
In an ideal world, every SEO-relevant page generated by faceted navigation should be a PLP with unique URLs, meta tags and content, but in some CMS types this is not possible. And that is why the Robot.txt file comes to our aid.
Use of robots.txt
The robots.txt file is a directory, located at the root of the website, that allows us to give directives to crawlers, such as Googlebot, when they crawl our website. Among the different directives we can add within the file, we have the ‘disallow’ command which allows us to indicate to the crawlers not to visit files, pages or entire folders of our website.
Going back to the faceted navigation example, robots.txt allows us to at least partially solve the problem by excluding the crawling of pages auto-generated by the filters.
In this way we can direct the resources of the crawl budget to more relevant subdirectories of the website.
Crawl Budget Monitoring
Optimising the crawl budget is an essential step to improve the visibility and performance of a website in search engines. However, optimisation alone is not enough; it is also important to constantly monitor how our crawl budget is allocated.
Once again, Google Search Console comes to our aid. Under Settings > Crawl Stats there is a section containing crawl data with a breakdown of all requests divided into types and outcome:
It is important to use the Crawl Stats data in Google Search Console to make any timely changes to our optimisation strategies.
Conclusion
In conclusion, a Crawl Budget optimisation strategy is a must in terms of SEO and search engine ranking of a website. Managing the Crawl Budget effectively means optimising the resources allocated by Googlebot to visit our website, increasing the perception of authority and relevance. This implies better distribution of ‘link juice’ and increased visibility in search engine results. To get the most out of our Crawl Budget, it is important to optimise site speed, structure, manage redirects and outdated pages, optimise the sitemap and manage faceted navigation. In addition, constant monitoring of the Crawl Budget goes through tools such as Google Search Console and is crucial for tangible results in the medium and long term.
