Deduplication

The problem

A wire story republished by forty outlets is forty near-identical passages. Rank them by relevance and they occupy every slot in your result set — ten results, one fact, and an agent that now believes the claim is extremely well supported because it saw it ten times.

collapse=story

Returns one representative passage per near-duplicate cluster, with group_size recording how many were folded in. Ten slots become ten distinct facts.

Set this by default for any query over news, finance, sport or press-release-driven content. It is worth more than any amount of ranking tuning.

curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/search?q=central+bank+decision&collapse=story&facets=true"

collapse=host and collapse=page

host returns at most one passage per domain, which stops a single site dominating results — useful when you want breadth of source rather than depth on one. page collapses multiple passages extracted from the same URL, which is what you want when the question is "which pages", not "which passages".

Deduplication does not destroy corroboration

The obvious worry with deduplication is that collapsing forty copies to one loses the fact that forty outlets carried it. It does not. When a near-duplicate is rejected at admission, its host is recorded against the surviving passage — so independent_sources still counts every asserting host, and corroborate returns both that count and the number of merged duplicates.