What is Freshness?
Verified 5 August 2026
The web changes at roughly one to five percent a day, so keeping an index current is a matter of detecting change rather than recrawling volume. Conditional requests make unchanged pages nearly free — a 304 response costs a round trip and no content.
Sitemaps, feeds and last-modified headers narrow it further, which is why a well-behaved crawler asks for those before it asks for pages.
How unlob handles this: UnlobBot
Common questions
What is Freshness?
Freshness is how current an index is relative to the live web, maintained by recrawling changed pages rather than everything.
How does Freshness work in practice?
Sitemaps, feeds and last-modified headers narrow it further, which is why a well-behaved crawler asks for those before it asks for pages.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.