What is Near-duplicate detection?
Verified 5 August 2026
The web is enormously redundant: syndicated articles, mirrored documentation, scraped copies. Exact hashing misses all of it because a single differing character changes the hash, so near-duplicate detection uses locality-sensitive hashing or embedding similarity.
Detection is only half the job. Discarding duplicates naively destroys the evidence that many outlets carried the same claim — which is exactly the signal you need to judge it. Recording the rejected copy's host against the survivor keeps the index compact and the corroboration signal intact.
Common questions
What is Near-duplicate detection?
Near-duplicate detection identifies documents with substantially the same content despite differing in wording, formatting or boilerplate.
How does Near-duplicate detection work in practice?
Detection is only half the job. Discarding duplicates naively destroys the evidence that many outlets carried the same claim — which is exactly the signal you need to judge it. Recording the rejected copy's host against the survivor keeps the index compact and the corroboration signal intact.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.