What is Crawl-on-miss?
Verified 5 August 2026
Rather than pre-emptively crawling everything, record the gap and go and get it. The index improves precisely where people are looking rather than where a crawl schedule happened to point.
The economic property is that cost tracks demand rather than corpus size, which is a materially different curve from crawling the web speculatively and hoping the coverage is useful.
How unlob handles this: UnlobBot
Common questions
What is Crawl-on-miss?
Crawl-on-miss triggers a fetch when a query finds nothing adequate, so the index fills gaps where there is demonstrated demand.
How does Crawl-on-miss work in practice?
The economic property is that cost tracks demand rather than corpus size, which is a materially different curve from crawling the web speculatively and hoping the coverage is useful.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.