Glossary
What is Crawl-on-miss?
Verified 21 September 2026
Rather than pre-emptively crawling everything, record where queries came up short and let that steer the crawl. The index improves where people are looking rather than where a fixed schedule happened to point. The economic property is that cost tracks demand rather than corpus size.
It is easy to mistake for an on-demand fetch, and in unlob it is not one. A query that misses is answered from what the index holds now: ground returns stale or insufficient, and why_not says what is absent. Nothing is fetched while your call waits, and /describe reports on_demand_crawl as false so an agent can plan around it. A gap that recurs informs what the crawler does next, on its own schedule — so a missing source is something to raise in an evaluation, not something a query can summon.
How unlob handles this: UnlobBot
Common questions
What is Crawl-on-miss?
Crawl-on-miss is a crawl policy in which a query that finds nothing adequate is recorded as a gap, and the gap steers what gets fetched next.
How does Crawl-on-miss work in practice?
It is easy to mistake for an on-demand fetch, and in unlob it is not one. A query that misses is answered from what the index holds now: ground returns stale or insufficient, and why_not says what is absent. Nothing is fetched while your call waits, and /describe reports on_demand_crawl as false so an agent can plan around it. A gap that recurs informs what the crawler does next, on its own schedule — so a missing source is something to raise in an evaluation, not something a query can summon.
See it working
The free tier is 10,000 credits a month with no card — enough to test any of this against your own queries.