What is Crawl budget?
Crawl budget is the number of URLs a search engine is willing and able to crawl on your site within a given period.
Google describes crawl budget as a mix of crawl capacity, how much crawling your server can handle without slowing down, and crawl demand, how much Google wants to crawl based on popularity and freshness.
Most small and medium sites do not need to worry about it. It matters for large sites with tens of thousands of URLs, fast changing inventory or lots of auto generated pages.
What affects crawl budget
Google defines crawl budget as the combination of crawl capacity (how much your server can handle) and crawl demand (how much Google wants to crawl). It mainly matters for large sites with tens of thousands of URLs or more, or sites that add many pages quickly.
| Factor | Raises crawling | Lowers crawling |
|---|---|---|
| Server speed | Fast, stable responses | Slow responses, 5xx errors |
| Popularity | Pages with links and traffic | Unlinked, low value pages |
| Freshness | Regularly updated content | Stale pages |
| URL waste | Clean URL structure | Parameters, filters, duplicates, redirects |
How to check crawl activity
- 1Open Settings, then Crawl stats in Google Search Console.
- 2Review total requests, average response time and response codes.
- 3Check which file types and Googlebot types are crawling most.
- 4Analyse server logs to see exactly which URLs are crawled.
- 5Compare crawled URLs with the pages you actually want indexed.
How to make better use of crawl budget
Cut duplicate and parameter URLs
Use canonicals, consistent internal links and robots.txt rules for useless filters.
Fix redirect chains and errors
Every hop and error wastes a request.
Faster server response times
Faster responses let Google crawl more pages.
Keep sitemaps clean
List only canonical, indexable URLs with accurate lastmod dates.
Prune thin pages
Merge or remove low value pages that will never rank.
Who needs to worry about crawl budget
Large sites
Sites with hundreds of thousands of URLs, such as big stores and listing sites.
Sites with many parameters
Filters and tracking codes that create endless URL variations.
Most small sites do not
A site with a few hundred pages rarely has crawl budget problems.
What crawl budget means
Crawl budget is the number of URLs a search engine is willing and able to crawl on a site in a given period. It depends on crawl capacity, which reflects how fast and stable the server is, and crawl demand, which reflects how popular and fresh the pages are.
Most small sites never need to worry. Crawl budget matters for large sites with tens of thousands of URLs, sites that generate many parameter URLs and sites with slow servers.
Crawl budget optimisation steps
- 1Review crawl stats in Search Console.
- 2Find low value URLs being crawled, such as filters and session IDs.
- 3Block or remove useless URL patterns.
- 4Fix redirect chains and broken links.
- 5Improve server response times.
- 6Keep sitemaps clean and current.
- 7Link to important pages prominently.
Crawl budget signals
Server speed
Faster responses allow more crawling.
Popularity
Pages with more links and traffic are crawled more.
Freshness
Frequently updated pages are revisited more often.
Crawl budget checklist
- 1Crawl stats reviewed.
- 2Parameter URLs controlled.
- 3No redirect chains.
- 4Fast server responses.
- 5Clean sitemaps.
Server logs and crawl budget
Server log analysis shows exactly which URLs bots request and how often. It reveals wasted crawling on filters or old URLs that tools and Search Console reports may miss.
Why Crawl budget matters for SEO
When crawl budget is wasted on low value URLs, important new or updated pages can take a long time to be discovered and indexed.
Improving crawl efficiency can speed up indexing of new products, articles and price changes, which matters for ecommerce, news and listings sites.
How Crawl budget works
Crawl budget improves when you remove waste and speed up the site.
Cut low value URLs
Control filters, parameters, internal search pages and duplicate URLs.
Speed up the server
Faster responses let Google crawl more pages without strain.
Strengthen signals
Link important pages internally and keep sitemaps clean and current.
A property listings site has 3 million URLs from search filters but only 80,000 active listings. Blocking filter URLs and cleaning the sitemap shifts crawling to real listings, and new properties appear in Google within a day instead of two weeks.
Crawl budget checklist
- ✓Check crawl stats in Search Console
- ✓Block or control low value parameter URLs
- ✓Fix redirect chains and broken links
- ✓Keep sitemaps limited to indexable URLs
- ✓Improve server response times
- ✓Remove or noindex thin auto generated pages
Common Crawl budget mistakes
Worrying about it on small sites
Sites with a few hundred pages rarely have crawl budget issues.
Using noindex to save budget
Noindexed pages still get crawled.
Endless calendar or filter URLs
These can trap crawlers.
Slow servers
Slow responses reduce how much Google crawls.
Related SEO terms
Crawl budget FAQs
What is crawl budget?
+
The number of URLs a search engine crawls on your site in a set period.
How do I check crawl activity?
+
Use the Crawl Stats report in Search Console and server logs.
Do redirects waste crawl budget?
+
Chains and loops do. Single redirects are fine.
Does noindex save crawl budget?
+
Not fully. Google still crawls noindexed pages, though less often over time.
Do large images affect crawl budget?
+
Slow, heavy pages reduce how quickly bots can crawl, so compressing images helps on large sites.
Does crawl budget affect rankings?
+
Not directly. It affects how quickly pages are discovered and refreshed, which matters on large sites.
Does my site need to worry about crawl budget?
+
Usually only if it has many thousands of URLs or changes very often.
Does site speed affect crawl budget?
+
Yes. Faster servers allow more crawling.
Can I ask Google to crawl more?
+
You cannot set a budget, but improving speed and removing waste helps.
Do AI crawlers use crawl budget?
+
AI crawlers consume server resources too. Monitor server logs to see their activity.
Can internal search pages waste crawl budget?
+
Yes. Block internal search result URLs from crawling unless they have unique value.
How do I increase crawl rate?
+
Improve server speed, fix errors, earn links and update content regularly.