Trust & accuracy
Measuring an index before you depend on it
Every vendor in this category says its index is comprehensive, and none of them can be wrong, because the claim has no denominator. The useful evaluation is not comprehensiveness. It is whether a specific index holds the specific things your product asks about, how fast it notices when those things change, and what it tells you when it comes up empty.
Verified 21 September 2026
The measurement, end to end
Derive the seed list
Pull 200-500 URLs from your own citation and retrieval logs, and tag each with the stratum whose absence would cost you most.
Add controls
Include deleted pages, login-walled pages and hosts that disallow crawling, so you can tell a coverage gap from a correct refusal.
Probe presence per URL
Ask the index about the URL rather than running your queries, so coverage and ranking stay separable.
Classify the misses
Split never-crawled, removed-with-reason, robots-blocked and bad-seed. Only the first two are the vendor.
Measure lag on a second sample
Twenty predictable publishers, probed daily, reported as a distribution rather than a mean.
Report per stratum and re-run
Share present, share explained, median lag. Keep the presence call behind one function so a rival can be measured with the same harness.
Why a coverage percentage is meaningless
A coverage figure needs a denominator, and the open web does not have one. There is no census of pages, no agreed definition of which of them count, and no way to distinguish a page that does not exist from a page nobody has ever linked to. Any vendor quoting a percentage has chosen a denominator that flatters it, and you cannot check which.
Page counts are no better. An index of a hundred billion URLs that holds none of the fourteen standards bodies your compliance product cites is worse for you than an index a tenth the size that holds all fourteen. Size is a property of the index; coverage is a relation between the index and your workload, and only you have the second half.
So stop trying to evaluate the index and start evaluating the fit. The rest of this guide is one way to do that which takes an afternoon and produces a number you can re-run against a second vendor without rewriting anything.
Build the seed list from what you already depend on
The seed list is the whole method. Get it from your own logs rather than from intuition: the URLs your agent has cited in the last quarter, the domains your analysts paste into tickets, the sources your existing retrieval returns most often. Two to five hundred URLs is plenty, and they should be the ones whose absence would be noticed.
Stratify them. Group by the thing that would make a miss expensive — publisher tier, content type, language, and age. A flat list gives you one number; a stratified one tells you that an index is strong on documentation and weak on regional press, which is the finding that actually changes a decision.
Include some URLs you expect to be absent. Pages behind logins, pages that were deleted, pages on hosts that disallow crawling in robots.txt. These are the control group: an index that claims to have them is telling you something about how it answers, and an index that explains why it does not is telling you something more valuable than a hit.
# One URL per line, with the stratum you care about in the second column.
cat > seeds.tsv <<'EOF'
https://www.rfc-editor.org/rfc/rfc9110.html standards
https://docs.python.org/3/library/asyncio.html docs
https://www.federalregister.gov/documents/2026/03/04 regulatory
https://example.invalid/deleted-last-year control-absent
EOF
wc -l seeds.tsvProbe presence, not ranking
The instinct is to run your real queries and see whether the right page comes back first. That measures ranking and coverage at the same time and cannot separate them, so a bad result tells you nothing about which to fix. A page the index holds but ranks twelfth is a ranking problem you can solve with filters. A page the index does not hold is a coverage problem no parameter will fix.
Ask about the URL directly instead. On unlob that is why_not, which takes an absolute URL and returns present, removed with a reason, or unknown — never an empty result set you have to interpret. On providers without a presence endpoint, the closest approximation is a site-scoped search for an exact salient string from the page, which is noisier but workable.
Record the raw answer per URL rather than a running total. You want to be able to go back and ask which stratum the misses were in, and a counter cannot answer that.
while IFS=$'\t' read -r url stratum; do
status=$(curl -sS -H "x-api-key: $UNLOB_API_KEY" \
--get --data-urlencode "url=$url" \
"https://api.unlob.com/why_not" | jq -r '.status + " " + (.reason // "-")')
printf '%s\t%s\t%s\n' "$stratum" "$status" "$url"
done < seeds.tsv > coverage.tsv
cut -f1,2 coverage.tsv | sort | uniq -c | sort -rnClassify every miss, because they are not the same miss
A missing URL has at least four distinct causes, and they carry completely different costs. Never crawled means the index has not reached it and might if it were asked — on unlob a query that misses schedules a fetch, so a second probe a day later is a different experiment rather than a repeat of the first. Removed means it was held and dropped, and the reason matters: superseded is healthy compaction, stale is a freshness signal, access-starved means the host made fetching unreliable.
Blocked by robots.txt is not the index failing. It is the publisher declining, and no vendor will beat it — if a miss is in this category, the fix is a licensing conversation or a different source, not a different search API. Checking this yourself before you blame the index takes one request per host and removes a surprising share of apparent gaps.
The fourth cause is your seed list being wrong: a URL that has moved, a canonical you recorded rather than the one the publisher serves, a tracking parameter that makes the URL unique to you. Expect this to be five to ten per cent of any list assembled from logs, and audit the misses by hand once rather than building a pipeline that carries the error forward.
Measure the lag separately from the coverage
Coverage and freshness are independent failures and a single pass conflates them. An index can hold ninety per cent of your seed list and still be useless for a monitoring workload if it sees changes eleven days late, and the coverage probe above will report that index as excellent.
Measure lag with a different instrument: pick twenty sources that publish on a predictable cadence, record the publication timestamp the publisher states, and probe daily until the passage appears. The distribution matters more than the mean — a median of six hours with a long tail of two weeks is a different product from a flat two days, and only one of them can back an alerting feature.
Be careful which clock you use. Crawl time and publication time are different fields for a reason; bounding by the fetch clock when you meant the content clock is the single most common mistake in this measurement, and it makes a fast index look slow.
Turn it into something you can re-run against a rival
The output should be a small table: per stratum, the share present, the share explained, and the median lag. Explained is its own column because it is its own capability — an index that misses fifteen per cent of your standards documents and tells you which fifteen is operationally different from one that misses twelve per cent silently, and on most regulated workloads the first is the better buy.
Keep the script provider-agnostic by isolating one function: given a URL, return present, absent-with-reason, or unknown. Everything else in the harness is arithmetic. Swapping vendors is then a twenty-line change rather than a rewrite, which is the only reason the measurement gets re-run at renewal time rather than done once and forgotten.
Run the whole thing on a free tier before you talk to anyone about price. On ours, 10,000 credits a month covers a 500-URL seed list 20 times over at one why_not credit per URL, which is enough for the initial pass and several re-runs.
What this method will not tell you
It will not tell you whether the index is good at retrieval. Presence is necessary and nowhere near sufficient: an index can hold every document you asked about and still rank the wrong passage first on every query. Ranking quality needs a labelled query set and human judgement, which is a larger project than this one.
It will not generalise beyond your seed list. That is the point — the number is deliberately about your workload — but it means you cannot quote it as a statement about the index in general, and you should not let a vendor quote it back at you that way either.
And it will not survive a change in your product. When the workload moves into a new domain or a new language, the seed list is stale and the number is decorative. Re-derive it from logs at the same cadence you review anything else load-bearing.
Frequently asked questions
How many URLs does the seed list need?
Two to five hundred is usually enough to separate strata that differ materially. Precision beyond that is spent on a number whose denominator you chose anyway, and the effort is better spent stratifying the list you have.
Can I run this against a provider that has no presence endpoint?
Yes, with more noise. Search for an exact salient string from the page scoped to its host, and treat a hit as present. You lose the ability to distinguish never-crawled from removed, which is the part of the answer most worth having.
Does a miss mean the page will never be returned?
Not necessarily. On unlob a query that misses queues a fetch, so a URL reported unknown today may be present tomorrow. Probe twice, a day apart, before you record a permanent gap.
Should I include the vendor in the comparison at all if it scores well on price?
Price is the easiest thing to compare and the least likely to be the reason a deployment fails. Run the coverage measurement first, drop anything that misses a stratum you cannot work around, and compare price among what is left.
Try it against your own queries
10,000 free credits a month, no card. Everything in this guide works on the free tier.