Guide Google to pages you want to rank, index them faster and keep them updated. Bring seasonal pages into focus to increase impressions.
page importance is the idea that there are measurable ways to determine which pages to prioritize.
There’s no index of set values of crawls for each site. Instead, available crawls are distributed based on what Google thinks your server will handle and the interest it believes users will have in your pages.
Your website’s crawl budget is a way of quantifying how much Google spends to crawl it, expressed as an average number of pages per day.
Why optimize your crawl budget?
Thanks to OnCrawl’s data on hundreds of millions of pages, we’ve also learned that there is a strong correlation between how frequently Google crawls a page and the number of impressions it receives: pages that are crawled more often are seen more often in search results.
Relation between number of impressions and crawl frequency
This correlation means that you can use crawl budget optimization as a strategy to promote a group of pages in search results. If your website has seasonal pages, these pages can be excellent candidates for promotional campaigns based on optimized crawl frequency.
To bring these pages to the forefront in search results, you’ll need to promote them to Google above other types of pages in your website during the appropriate seasonal period.
Using crawl budget optimization strategies, you can draw Google’s attention to certain pages and away from others in order to increase impressions on pages subject to seasonality on your website.
You’ll want to:
- Optimize your general crawl budget.
- Reduce the depth of important season pages using “collections” linked to from category home pages in your site structure.
- Increase the internal popularity of important pages by creating backlinks from related pages.
Relation between the number of internal “follow” links and crawl frequency
official 12 bots, at OnCrawl we’ve noticed a new bot emerging: Google AMP bot. This data includes all URLs — including JavaScript, CSS, font and image URLs — for all bot hits. Because of differences in bot behavior, the values given are averages. For example, since AdSense and mobile bots must fully render each page, unlike the desktop Googlebot, the page load time provided is an average between the full and partial load times.
This isn’t precise enough for SEO analyses.
Therefore, the most reliable way to measure your site’s crawl budget is to inspect your site’s server logs regularly. If you’re not familiar with server logs, the principal is straightforward: web servers record every activity. These logs are usually used to diagnose site performance issues.
One activity logged is the request for a URL. In the log, lines for this type of activity will include information about the IP address making the request, the URL, the date and time and the result in the form of a status code.
Here’s an example:
www.mywebsite.com:443 66.249.73.156 [15/Aug/2018:00:02:59 +0000] “GET /news/my-article-URL HTTP/1.1” 200 44506 “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
By identifying all of the requests from search Google bots, you can accurately measure the number of Google bot hits in a given period of time. This is your crawl budget.
Evidence from combining log data and semantic analysis by OnCrawl supports this fact. We’ve found most sites show a relationship between:
- Number of words and crawl behavior.
- Duplicate content and crawl behavior.
- Internal PageRank and crawl behavior.
You should also leverage the advantage of quality content to reinforce weaker pages through the use of:
- External backlinks.
- Internal linking structures.
- Canonical optimization.
If you’re promoting seasonal pages, concentrate on optimizing them first. Reports from site audits and site crawls indicate which pages in these groups would profit most from improvement.
Your healthy crawl budget
A healthy crawl budget is the key to improving ROI on SEO efforts by ensuring that Google sees the pages you’ve optimized.
Once you’ve made improvements, continue monitoring the crawl budget of your site. This allows you to measure the results and be ready to react to changes.
