Crawl budget
The number of URLs a crawler will fetch from a site in a given period.
SEO
Definition
Crawl budget is the practical limit on crawling, set by a site's capacity to serve requests and the crawler's assessment of how much the content is worth re-fetching.
It becomes a real constraint on large sites with faceted or generated URLs.
Why it matters
Wasted crawl on parameters and duplicates delays discovery of the pages that earn revenue.
How to apply it
- Consolidate duplicates with canonicals and clean internal linking.
- Keep the sitemap to indexable, canonical URLs only.
- Fix slow responses before adding more URLs.