Stop agents hallucinating from search results

Failure one: repetition read as corroboration

A wire story republished by forty outlets produces forty near-identical documents. Ranked by relevance they occupy every slot in the result set, and an agent that sees the same claim ten times concludes it is exceptionally well supported.

It is exactly backwards. One claim from one source, copied, is weaker evidence than three independent reports — and the ranked list presents it as stronger.

The fix has two halves. Collapse by story so one item cannot occupy ten slots, and filter on the number of distinct hosts asserting it. The subtlety is that these must be designed together: naive deduplication destroys the corroboration signal, because the copies you discarded were the evidence.

curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/search?q=acquisition+announced&collapse=story&min_independent_sources=3"

# One representative per story cluster, and only stories carried
# by at least three distinct hosts.

Verifying a specific claim

When the agent has a passage and needs to know how well supported it is, corroborate returns the source groups directly: how many distinct hosts, which ones, and how many duplicates were merged into the survivor.

That last number is informative on its own. A story with two independent sources and thirty merged duplicates is a press release; two independent sources and no duplicates is two newsrooms that did the work.

curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/corroborate?id=p:8f2c9a"

{ "independent_sources": 7, "merged_duplicates": 23,
  "sources": [{ "host": "reuters.com", "host_rank": 0.94 }, ...] }

Failure two: absence read as non-existence

An empty result set means one of three things: the page does not exist, it was never crawled, or it was indexed and removed. An agent has to pick, and it will pick confidently.

This produces a specific and common failure: "there is no documentation for that parameter" when the documentation exists but was not indexed. The agent is not inventing anything — it is drawing the only conclusion its evidence supports.

Giving the agent a tool that answers the question removes the inference. Add why_not to the tool list and mention it in the system prompt.

@function_tool
def why_not(url: str) -> dict:
    """Explain whether a URL is in the index, was removed and why,
    or was never seen. Use this before concluding something
    does not exist."""
    r = requests.get(f"{BASE}/why_not", headers=HEADERS,
                     params={"url": url}, timeout=10)
    r.raise_for_status()
    return r.json()

Make the selection explainable

When an answer has to be defended, "the model chose these sources" is not an account of anything. assemble_context returns a reason with every included passage — corroborated by N hosts, highest centrality in its cluster, most recent authoritative source.

That turns the retrieval step into something you can show someone, which for regulated work is frequently the requirement rather than a refinement.