Quick answer: Crawl budget is the set of URLs Googlebot can and wants to crawl on a website in a given period, decided by the site’s crawl capacity limit (how much load the server can handle) and crawl demand (how much Google wants its content).
Google does not crawl every URL on your site every day. Googlebot has limited time and server connections, and the web is large, so it decides how many of your URLs to fetch and which ones to fetch first. That allowance is your crawl budget.
Two things set it. Crawl capacity limit is how hard Googlebot can hit your server without slowing it down: a fast, healthy server gets more requests, a slow one or one returning errors gets fewer. Crawl demand is how much Google wants your pages, driven by how popular they are, how often they change, and how many URLs Google thinks you have. A page that is never crawled cannot be indexed, and a page that is not indexed cannot rank or be cited by AI Overviews.
Why crawl budget matters (and when it does not)
Crawl budget is a real constraint for large or fast-changing sites, and a non-issue for most small business websites. Google says its crawl budget guide is meant for sites with roughly 1 million or more unique pages that change about weekly, sites with 10,000 or more pages that change daily, or sites with a large share of URLs stuck as “Discovered, currently not indexed” in Search Console. Google also says its Crawl Stats report is aimed at advanced users and that sites with fewer than a thousand pages usually do not need it.
That honesty matters. If you run a 40-page services website in Mumbai and a page is not ranking, crawl budget is almost never the reason. Content quality, internal links and search intent are. Do not let anyone bill you for “crawl budget optimisation” on a small brochure site.
Where it does bite is ecommerce and listing sites. Filters, sort orders, session IDs and tracking parameters can turn 2,000 products into tens of thousands of crawlable URLs. Googlebot then spends its visits on near-duplicate filter pages while your new collection, sale page or price change waits in the queue. For a D2C brand running a Diwali sale, a sale page that gets indexed after the sale ends is revenue you never see.
One myth to drop: more crawling does not mean better rankings. Google states that crawling is required to appear in results but is not a ranking signal.
Crawl capacity limit vs crawl demand
| Attribute | Crawl capacity limit | Crawl demand |
|---|---|---|
| What it measures | How much crawling your server can handle | How much Google wants to crawl your URLs |
| Main drivers | Server response time, uptime, 5xx errors, 429 rate-limit responses | Page popularity, how often content changes, how many URLs Google thinks exist |
| Raised by | Faster hosting, caching, fewer server errors | Useful content, strong internal links, fresh updates that matter |
| Lowered by | Slow responses, timeouts, frequent 5xx errors | Duplicate pages, thin pages, URLs Google sees as low value |
| Who controls it | Your developer and hosting provider | Your SEO and content team |
How to check and improve crawl budget
The goal is not a bigger crawl budget, it is a cleaner one: every Googlebot visit should land on a page you actually want indexed.
- Check if you even have a problem. In Google Search Console, open Indexing, then Pages. If a large share of important URLs sit under “Discovered, currently not indexed,” crawling may be the bottleneck. If not, look elsewhere first.
- Open the Crawl Stats report. Go to Settings, then Crawl stats (available on root-level properties). Look at total crawl requests, average response time, host status, and the split of responses by type. Most responses should be 200 OK.
- Find where crawls are wasted. Crawl your own site with a tool like Screaming Frog or Sitebulb and compare the URL count with the pages you actually want indexed. Filter and sort parameters, internal search results, session IDs and printer-friendly versions are the usual leaks.
- Block low-value URLs in robots.txt. Disallow faceted filter combinations, internal search pages and similar URL patterns you never want in search. Google warns against using robots.txt to shuffle budget around temporarily, so only block what should stay blocked.
- Return proper status codes for dead pages. Use 404 or 410 for permanently removed pages, and fix soft 404s (empty pages that return 200). Google calls a 404 a strong signal not to crawl that URL again.
- Shorten redirect chains. Point old URLs straight to the final page with one 301. Every extra hop is another request Googlebot spends.
- Keep XML sitemaps clean and current. List only indexable, canonical URLs, and include an accurate lastmod date for pages that changed.
- Speed up the server. Better hosting, server-side caching and support for 304 Not Modified responses let Googlebot fetch more pages in the same time.
Worked example: a WooCommerce store with filter bloat
This is an illustrative scenario, not client data. A Mumbai fashion brand has 2,000 product pages. Its category pages have filters for size, colour, price and fabric, and each combination creates a new URL. A site crawl finds about 60,000 crawlable URLs.
- Crawl Stats shows about 3,000 Googlebot requests a day.
- If 70% of those requests go to filter URLs: 3,000 x 0.70 = 2,100 wasted requests a day.
- That leaves about 900 requests a day for the 2,000 products, categories and new pages that actually earn revenue.
- After blocking filter combinations in robots.txt and cleaning the sitemap, most of those 3,000 daily requests can go to pages that sell.
The fix cost no ad spend. It simply stopped Google from spending its visits on pages no customer ever searches for.
Common mistakes
- Using noindex to save crawl budget. Google still has to request a noindex page to see the tag, so it uses crawl time anyway. Use robots.txt for URLs that should not be crawled, and noindex for pages that can be crawled but should not appear in search.
- Blocking a page in robots.txt and adding noindex to it. If Googlebot cannot crawl the page, it never sees the noindex tag. Pick one method based on what you want.
- Worrying about crawl budget on a small site. For sites with a few hundred pages, Google crawls what matters. Fix content and internal linking before touching crawl settings.
- Adding crawl-delay to robots.txt. Google’s crawlers do not process the crawl-delay rule. If Googlebot is overloading your server, fix server capacity or follow Google’s guidance on reducing crawl rate.
- Faking freshness. Changing dates or making trivial edits to trigger recrawls does not help. Google judges content on quality, and needless URL changes waste crawl resources.
Frequently asked questions
What is crawl budget in SEO?
Crawl budget is the set of URLs Googlebot can and wants to crawl on your site in a given period. It depends on your crawl capacity limit, which is how much load your server can handle, and crawl demand, which is how much Google wants your content.
Does crawl budget matter for small websites?
Usually not. Google’s crawl budget guide is aimed at sites with around 1 million or more pages, or 10,000 or more pages that change daily, and Google says sites with fewer than a thousand pages usually do not need the Crawl Stats report. Small sites should focus on content quality and internal links instead.
Is crawl budget a Google ranking factor?
No. Google states that crawling is necessary for a page to appear in search results but is not a ranking signal. Getting crawled more often does not by itself improve rankings.
How do I check my crawl budget in Google Search Console?
Open Settings and then Crawl stats in Google Search Console. The report shows total crawl requests, total download size, average response time, host status, and crawl responses broken down by status code, file type, purpose and Googlebot type.
Does noindex save crawl budget?
No. Googlebot still requests a page with a noindex tag and only drops it after reading the tag, so the crawl time is still spent. To stop crawling of low-value URLs, block them in robots.txt or remove them and return a 404 or 410.
How Uginix helps
Uginix checks crawl health as part of every technical SEO audit, but only treats crawl budget as a priority when the site is large enough for it to matter, usually ecommerce or listing sites with filter-heavy navigation. For smaller sites, we tell you plainly that the gains are elsewhere. Explore our SEO and performance marketing services, read how search is shifting in GEO vs SEO, or ask for a free growth audit, no lock-in, no pitch deck.
Related terms
- Redirect loop: an error where redirects circle back to an earlier URL, so the page never loads for visitors or crawlers.
- Redirect chain: a series of redirects that eventually reaches a working page, using an extra crawl request at every hop.
- Robots.txt: a file at the root of a site that tells crawlers which URLs they may and may not crawl.
- XML sitemap: a file listing the URLs you want search engines to discover and crawl.
- Canonical URL: the preferred version of a page you tell search engines to index when duplicates exist.
Sources: Google Ads Help, About auto-tagging, Measure School, UTM parameters in GA4, Optimize Smart, GA4 UTM parameters not working, Infinity Group, UTM tagging in GA4.