What is Deduplication?
Verified 5 August 2026
At index time it prevents the same content occupying storage many times. At query time it prevents one item occupying every slot in a result set.
The trap is that deduplication discards information about how widely something was published — which for judging a claim is the most valuable information available. Deduplication and corroboration counting have to be designed together, or you get a compact index that cannot tell you whether anyone corroborated anything.
Common questions
What is Deduplication?
Deduplication removes redundant copies of the same content from an index or a result set.
How does Deduplication work in practice?
The trap is that deduplication discards information about how widely something was published — which for judging a claim is the most valuable information available. Deduplication and corroboration counting have to be designed together, or you get a compact index that cannot tell you whether anyone corroborated anything.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.