Getting recency right in agent retrieval

Four clocks, routinely conflated

The publication clock is what the page says about itself: a dateline, a structured-data field, a byline date. It is the one people mean by recent, and it is self-reported, which means it is sometimes wrong and frequently absent.

The crawl clock is when the index fetched the page. It is reliable because we control it, and it answers a different question: what evidence existed at this point in time. That is an audit question — reproducing a past result set, or defending what was considered — and it is the wrong instrument for currency, because a page fetched this morning may have been written in 2019.

The index clock is when the passage became retrievable, which can lag the crawl clock. It matters for debugging ("why did my probe miss this an hour ago") and almost never for a user-facing query.

The fourth clock is the one no index holds: when the fact itself changed. A tutorial published in 2021 describing an API that has not changed since is current. A market figure published this morning citing last quarter is not. No date filter distinguishes these, and pretending otherwise is where the confident wrong answers come from.

Choosing between the bounds

On unlob, published_from and published_to bound by the content date and from and to bound by the crawl time. Defaulting to the publication pair is right for almost every user-facing question; reaching for the crawl pair is right when you are reconstructing what an earlier run would have seen.

The trap is that both accept the same kind of value and neither will complain. A query bounded by the fetch clock when you meant the content clock returns pages crawled recently, which on a freshly-expanded index is an arbitrary slice of the corpus rather than the recent web. It looks like it works, and it is wrong in a way no error surfaces.

Write the intent down where the query is built. A comment saying "content date, because the user asked what is current" costs nothing and survives the next person to touch the code, who will otherwise reach for whichever parameter name they remember first.

# What the user means: published in the last 90 days.
curl -H "x-api-key: $UNLOB_API_KEY" --get \
  --data-urlencode "q=capital requirements guidance" \
  --data-urlencode "published_from=$(date -u -d '90 days ago' +%s)" \
  "https://api.unlob.com/search"

# What an auditor means: whatever the index held during Q1.
curl -H "x-api-key: $UNLOB_API_KEY" --get \
  --data-urlencode "q=capital requirements guidance" \
  --data-urlencode "from=1767225600" --data-urlencode "to=1774915200" \
  "https://api.unlob.com/search"

Hard bound or soft preference

A hard bound removes everything outside the window, including every page that carries no parseable publication date. That exclusion is the conservative behaviour and it is the correct default for a filter that claims to bound time — but it narrows the pool more than most people expect, because a large share of the web publishes no reliable date at all. Documentation is a notable offender, and documentation is often exactly what the agent needed.

prefer_recent is the soft alternative: newer passages rank higher, nothing is removed. Use it when older material may still be valid and you would rather demote it than lose it — which describes most technical questions and almost all how-to questions.

The decision rule is about the cost of the two errors. If returning something stale is worse than returning nothing, bound hard. If returning nothing is worse than returning something a bit old, bias soft. Teams that reach for the hard bound by reflex usually discover the miss rate later, in the form of an agent that says it could not find anything about a subject the index covers well.

Where recency and deduplication fight

News is the worst case. A wire story is republished by dozens of outlets, and the republished copies frequently carry a later timestamp than the original reporting. Sort by date and the top of the result set fills with the syndicated tail rather than the source — newer, identical, and less attributable.

Collapsing to one representative per story cluster fixes the symptom, and it is worth understanding which representative you get: the cluster keeps the corroboration count across all its members, so collapsing does not cost you the evidence that a claim was independently reported. You get one passage and an honest count rather than twelve passages and a misleading impression of consensus.

The related failure is timestamp churn on pages that are edited rather than republished. A page whose date field updates on every edit will keep re-entering a recency window without its substance changing, which inflates any count you build over that window. If you are monitoring rather than answering, deduplicate on content rather than trusting the date to tell you something happened.

curl -H "x-api-key: $UNLOB_API_KEY" --get \
  --data-urlencode "q=central bank rate decision" \
  --data-urlencode "published_from=$(date -u -d '7 days ago' +%s)" \
  --data-urlencode "collapse=story" \
  --data-urlencode "min_independent_sources=2" \
  "https://api.unlob.com/search"

The staleness no filter reaches

A date filter operates on what the index holds. It cannot tell you about a page published inside your window that the index has not fetched yet, and that gap is a property of crawl cadence rather than of the query. On a bounded index, where admission is a decision rather than an accident, the gap is smaller but it is never zero.

This is why a recency-sensitive workload needs the coverage question answered separately from the query. Probing whether a specific expected source is present — and, when it is not, whether it was never crawled or was dropped and why — is a different call from searching, and it is the one that tells you whether an empty recent window means nothing happened or means the index has not caught up.

Say which of the two it was. An agent that reports "no coverage in the last seven days" when it means "the index holds nothing from the last seven days" has made a claim about the world from a fact about a database, and that is the structural source of a whole class of confident errors.

A pattern that holds up

Start with the widest window the question tolerates and a soft preference rather than a bound. Look at the date distribution in the facets before deciding anything — a limit=0 probe returns counts and distribution with no results and costs almost nothing, and it usually shows the window you were about to impose was either far too narrow or entirely unnecessary.

Then tighten only if the distribution says you must, and pair the tightening with corroboration rather than with sorting. Requiring two independent sources inside a recent window is a much stronger guarantee than taking the newest single result, and it degrades more gracefully: you get fewer results rather than a plausible wrong one.

Finally, carry the dates through to the output. If the agent is going to assert something is current, the passage date should appear in the answer, so the reader can disagree with the inference. Most recency bugs survive because the date never reaches anyone who would have noticed it.

Where recency filtering makes results worse

Reference and standards material is the clearest case. The canonical document for a protocol may be a decade old and entirely current, and any recency bound will replace it with a blog post summarising it badly. For these questions, bias toward authority and centrality instead — what the field treats as foundational is a better proxy for correctness than what was written most recently.

Academic work behaves similarly but less absolutely. Recency genuinely matters for a fast-moving subfield and genuinely does not for an established method, and no single parameter encodes that difference. The practical compromise is a soft recency preference alongside a centrality floor, so recent work surfaces without displacing the paper everyone cites.

And for anything with a long tail of small publishers, a hard bound is disproportionately punishing, because small sites are the least likely to emit structured publication dates. You will filter out exactly the independent sources that made the corroboration count worth having.