Does Google Search Console show my crawl budget?
Technical SEO
No. Search Console has no crawl budget number, because crawl budget is not a fixed allowance. Google describes it as the set of URLs it can and wants to crawl on your site, set by how much crawling your server can take and how much Google wants your pages. Most sites never need to think about it: Google's guide on the subject is written for sites with over a million pages, or over 10,000 pages that change daily.
What decides your crawl budget
- Crawl capacity limit. How many connections Google opens and how long it holds them. Every site starts with the same conservative setting, which rises while your server responds quickly and falls with slow responses,
5xxerrors and429rate limiting. - Crawl demand. How much Google wants your URLs: how popular they are, how quickly they go stale, and how much of your URL inventory looks worth crawling. Duplicate and unwanted URLs waste demand. A site move raises it for a while.
Each hostname has its own budget, so www.example.com and shop.example.com are crawled separately.
Does your site need to care?
Google names three situations, and calls them rough estimates rather than thresholds:
- Sites with more than a million unique pages whose content changes about weekly.
- Sites with more than 10,000 unique pages whose content changes daily.
- Sites where a large share of URLs sit in Discovered – currently not indexed.
If your new pages are usually crawled on the day you publish them, crawl budget is not your problem. Pages that are not indexed are then far more likely held back by duplication, quality or configuration.
Where Search Console shows the warning signs
- Crawl stats (under Settings): falling requests with rising response times, host status problems, and the split between discovery and refresh crawling. See Crawl stats.
- Page indexing: a large or growing Discovered – currently not indexed count. See that reason explained.
- URL Inspection: a Hostload exceeded message means Google held back because your server was at capacity.
How to make crawling go further
- Consolidate duplicates, so Google spends its crawling on unique pages.
- Block URLs that should never be crawled in
robots.txt, such as sorted or filtered variants of the same list. - Return
404or410for pages that are gone for good, and fix soft 404s, which keep getting crawled. - Keep sitemaps current, with
<lastmod>for changed pages. - Avoid long redirect chains, and make pages fast to serve; supporting
304 Not Modifiedsaves bandwidth on unchanged pages.
Two common tactics do not work. noindex does not save crawl budget, because Google still requests the page to see the tag. And blocking pages in robots.txt does not hand their share to other pages unless your site is already at its capacity limit. To raise the budget itself, add server capacity when you see Hostload exceeded, and make the content worth crawling.
How AutoPSEO helps
The Index Tracker groups your tracked URLs by how long ago Google last crawled them, from within a week to over 120 days, and flags indexed pages left uncrawled for 120 days or more: the practical sign that a section is crawled less than you assume. AutoPSEO's content engine also keeps your URL inventory lean. Each planned topic is checked against your existing pages before anything is written, and two pages that answer the same searches can be merged, with the old address redirected to the one that stays. See topics in the content engine.
Sources: Google's crawl budget guide and Crawl stats report help.
More in Technical SEO
- How do I check mobile usability errors in Google Search Console?
- How do I check structured data in Google Search Console?
- How do I check my robots.txt in Google Search Console?
- How do I find and fix 404 errors in Google Search Console?
- What is the Page Experience report in Google Search Console?
- How do I check rich results in Google Search Console?
Put your SEO content on autopilot
Connect Search Console and your site. AutoPSEO scans it and plans the first topics right away. No credit card needed.