What is Crawl frontier?
Verified 5 August 2026
Managing the frontier is most of what a crawler does: deciding what is worth fetching next, when a host may next be contacted, and what to discard.
Prioritisation is where crawl budget is won or lost. Crawling breadth-first spends heavily on regions nobody queries; prioritising by demand spends where there is demand.
Common questions
What is Crawl frontier?
The crawl frontier is the prioritised queue of URLs a crawler intends to fetch, subject to per-host politeness constraints.
How does Crawl frontier work in practice?
Prioritisation is where crawl budget is won or lost. Crawling breadth-first spends heavily on regions nobody queries; prioritising by demand spends where there is demand.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.