Crawl budget is the set of URLs Google can crawl on a host and wants to crawl. Google's guide on managing it says you can skip the whole topic if your pages are crawled the day you publish them and you do not have a large set of URLs that change constantly. For Google Search, keeping the sitemap current and reading the Page indexing report is enough for everyone else. The rest of this article is for the sites the guide is actually aimed at, and for the publishers who have been told to worry anyway.
Google treats a site, for crawling, as one hostname. https://www.example.com/ and https://code.example.com/ have separate budgets. Fixing the www host does not fix the subdomain.
The two limits, in Google's words
Crawl capacity is how much fetching your server can absorb. Google calculates a host load so it does not hold too many connections open for too long. Every site starts from a conservative default. If the server stays fast and healthy and Google has a reason to fetch more, the limit rises. If responses slow down, or the host returns 5xx, or it returns 429, the limit falls and Google fetches less. 429 counts as a server error for this purpose. Capacity is also capped by Google's own finite resources, which are shared across the web.
Crawl demand is how much a given crawler wants from the host. Googlebot's demand depends on the site's size, how often pages change, and how useful and relevant those pages are compared with other sites. Three parts of demand are influenceable. Perceived inventory: Google will try to crawl the URLs it knows, so a pile of duplicates and junk URLs spends the visits you could have used elsewhere. Popularity: URLs that are widely linked get fetched again to stay fresh. Staleness: changed documents get fetched again so the change is seen. A site move can raise demand for a while because the new URLs have to be processed.
Capacity is shared by all Google crawlers on that host. A crawler with heavy demand can leave less room for the others. Even when the server could handle more, low demand means fewer fetches. Budget is capacity and demand together, not a dial in Search Console.
The guide gives rough sizes for who should care. They are estimates, not cutoffs: large sites on the order of a million unique pages that change about weekly, medium or larger sites on the order of 10,000 unique pages that change daily, and any site where a large share of known URLs sit in "Discovered – currently not indexed". If you are smaller than that and new URLs are fetched within a day, spend the week on the article and the links.
Waste is an inventory problem
The factor Google says you can control most is which URLs exist to be fetched.
Faceted navigation is the usual leak. Every combination of color, size, sort, and price is a new URL with the same products. Session identifiers and tracking parameters do the same thing to articles. Calendars that link to the next empty month forever, tag pages with no stories, and print views of every article add URLs that are not separate documents.
Google's recommended handling is to consolidate true duplicates, and to block crawling of the rest with robots.txt when you do not want them fetched at all. A noindex tag does not save the fetch: Google still requests the URL and drops it after seeing the tag. The guide is explicit that you should not use noindex to conserve budget, and that you should not use robots.txt as a temporary way to shove budget toward other pages. A disallow is for URLs you do not want crawled. Google will not move the freed capacity onto the rest of the host unless it is already hitting the capacity limit.
Removed pages should return 404 or 410, not a soft 404 and not a long-term disallow. A 404 is a strong signal to stop fetching that URL. A blocked URL stays in the queue and is fetched again when the block comes off. Soft 404s keep being fetched because they look like successful pages. The soft 404 guide is the status-code work. The robots.txt guide is the disallow work, including the warning that a disallow is a bad removal method.
Long redirect chains waste hops. Google generally follows up to 10. Link and canonicalise to the final URL so the chain is never crawled. The canonical guide keeps the preferred URL consistent so duplicates are not a second inventory.
Make the fetches you do want cheaper
If Google can load a page faster, it may get through more of the host. The guide points at two concrete habits. Keep server time-to-first-byte stable. Support 304 Not Modified so an unchanged URL does not send the whole body again. A 304 tells the next system the content matches the last crawl. For Google Search, signals may be recalculated, and the status is not an indexing penalty. It is a saving on bytes.
Do not look for a crawl-delay line to add. Google's robots.txt specification does not support that field. The host load mechanism is the rate limit. Slowing the server on purpose lowers capacity. It does not reserve a polite trickle for your favorite section.
Sitemaps belong in this list because they are how you name the inventory you care about. Keep them current, include lastmod only when it is real, and leave junk URLs out. The sitemap guide is that filter. A sitemap full of parameter URLs increases the perceived inventory. It does not concentrate the crawl.
How to tell whether you have the problem
Open Crawl stats in Search Console settings. You want the mix of response codes, the total requests, and whether response time is climbing. A host that answers 5xx or 429 is cutting its own capacity. A host that spends its requests on parameter URLs is wasting demand on inventory.
Open the Page indexing report and look at the size of "Discovered – currently not indexed" relative to the URLs you meant to publish. Filter by the article sitemap. If articles are indexed and the leftover is faceted URLs, you do not have an article problem. If the article sitemap itself is mostly discovered and not fetched, and the site is large or changing daily, look at capacity and at whether more important URLs are the ones being fetched. Reason-by-reason reading is the Page indexing guide.
If you have server logs, count Googlebot requests by path prefix for a day. When the top prefixes are /search, /tag/, or query-string variants, the leak is visible without a third-party crawl. When the top prefixes are the sections you care about, budget is not the reason a single story is missing. Inspect that story. The indexing diagnostic starts from the URL, which is the right scope for a small publisher.
URL Inspection can also show a host-load failure on a specific fetch. The crawl budget guide names "Hostload exceeded" in that tool as a sign you are at capacity. Adding servers is one of the two ways Google says budget grows. The other is demand: popularity, usefulness, uniqueness, and the ability to serve the pages. You cannot annotate your way to a larger budget with sitemap priority. Google ignores that field.
What to change, and what to leave
Block or stop generating the URL families that are not documents: internal search, endless sorts, session ids, empty calendars. Consolidate duplicates with redirects and canonicals instead of leaving both versions crawlable. Return 404 or 410 for pages you deleted. Shorten redirect chains. Speed the origin and honor 304 on unchanged URLs. Keep the sitemap equal to the set you want fetched.
Leave the site alone when new articles are fetched within a day and the discovered-not-indexed pile is small. Publishing another article will do more than a week of crawl tuning. Internal links still matter, because a URL nobody links to is a URL Google has less reason to want. That is demand, and it is the internal linking guide, not a budget trick.
Do not resubmit the sitemap in a loop, and do not request indexing on the junk URLs to "use the budget". Requests do not raise host load. They add specific URLs to a queue that is still judged afterward.
A sensible week for a large site
Day one: Crawl stats and a log sample, so you know whether the waste is capacity or inventory. Day two: one rule that stops the worst URL family, either by not linking it or by a precise robots.txt disallow, plus a check that article CSS and JavaScript stay fetchable. Day three: sitemap regeneration so those URLs are gone from the file, and 404s for anything you removed. Then wait through the robots.txt cache and the next round of fetches before you judge the chart. Changing three systems in one afternoon makes the next chart impossible to read.
If the chart was never bad, and a stakeholder still wants a crawl-budget project, show them the guide's first paragraph and go back to the articles that are crawled and not indexed. Those pages are a quality question. More server capacity will not answer it.
CrawlFaster can show indexing status by section, which is a practical way to see whether the URLs you meant to keep are the ones moving into the index after you cut the junk inventory. The host load and the URL families are still settings on your server, not a status you can toggle from a report.
