For architects
Benchmarks, with their conditions attached
Every number on this page carries the setup it was measured under. A benchmark without its conditions is marketing; the conditions are what let you decide whether the figure applies to your workload.
Verified 5 August 2026·Engine measurements
- Keyword p50
- 2.1 ms
- Hybrid p50
- 4.6 ms
- Serve QPS
- 471 / thread
- Languages
- 101
Single node, ~15k-passage synthetic corpus
BM25 fused with vectors, same corpus
Keyword mode, single thread
One embedding space, no translation step
Query latency
~15,000 admitted passages
| Mode | p50 | p95 | p99 |
|---|---|---|---|
| Keyword BM25 term ranking | 2.1 ms | 2.6 ms | 2.8 ms |
| Semantic Vector similarity | 3.3 ms | 3.9 ms | 4.3 ms |
| Hybrid Both, fused by reciprocal rank | 4.6 ms | 6 ms | 6.5 ms |
~31,000 admitted passages
| Mode | p50 | p95 | p99 |
|---|---|---|---|
| Keyword | 2.4 ms | 2.9 ms | 3.4 ms |
| Semantic | 4.5 ms | 5.6 ms | 6 ms |
| Hybrid | 6.1 ms | 7.5 ms | 8 ms |
Method.Measured single-node, single-thread against a synthetic five-vertical corpus of ~15,000 admitted passages across ~2,000 hosts.
A 2× corpus increase, from 15,000 to 31,000 admitted passages, moved keyword p50 by 0.3 ms in this benchmark. That is two measurements on a small corpus: evidence of the shape of the curve, not proof of it at a thousand times the size.
Throughput
What is measured here, and what is not
Everything on this page is a serving measurement: what the engine does with a request you send, timed at the engine. That is deliberate. Index-time rates and the memory layout of the serving tier describe how we built the thing rather than what it does for you, and we do not publish them.
The figures that remain are the ones you would actually use to size a workload — and unlike an architecture diagram, you can reproduce them. The free tier is 10,000 credits a month with no card, which is enough to run your own latency harness against your own queries from your own region.
Conditions, always attached
Single-node, single-thread, on a corpus we describe by size and shape. A benchmark without its conditions is marketing.
Engine time, not wall clock
Your network round trip is additional and, from most places, larger than the query itself. Budget for it before optimising anything here.
A small corpus
15,000–31,000 passages is not web scale, and we will not restate these as if it were. The scaling curve is evidence of a shape, not proof at a thousand times the size.
Agent grounding benchmark
Same model, same questions, different retrieval layer. What changes?
Milliseconds are the least of it. For an agent, the cost of retrieval is everything it has to do after the call: work out which results are copies, which sources are independent, whether it has enough, and what to read next. So the benchmark we will publish measures what happens to the agent — whether it reaches a defensible answer, two independent origins for every factual claim, and whether it says so when it cannot.
The headline metric is calibrated abstention. The task set includes claims whose evidence does not exist, and the right answer to those is "not found". A retrieval layer that always returns something plausible scores zero there, however good its ranking is.
Every provider is driven by the same fixed loop and graded by the same judge, so the only thing that differs between rows is the retrieval. For each claim the loop issues up to four queries in a fixed order and stops as soon as the evidence holds two independent supporting origins, or two contradicting ones. The task set mixes stable, well-documented facts, negated versions of the same facts, and invented entities whose evidence does not exist, so abstaining is sometimes the right answer. Two items count as one origin when they share a publisher, the same article title or near-identical text, using no provider's own ownership data. The harness, the task set and the provider adapters will be published alongside the first results, so anyone can rerun it with their own keys.
What each column will measure
| Metric | Definition | Better |
|---|---|---|
| Calibrated abstention | Of claims whose evidence does not exist, the share where the agent correctly said "not found" instead of asserting something. | higher |
| Task accuracy | The verdict matches the expectation: supported claims supported, contradicted claims contradicted, absent claims abstained on. | higher |
| Unsupported-claim rate | Of claims the agent asserted, the share that should not have been asserted. | lower |
| False-absence rate | Of claims that are supported on the web, the share where the agent gave up and abstained. | lower |
| Two-origin rate | Of supported claims, the share that reached two independent supporting origins. | higher |
| Independent-origin recall | Supporting publishers a provider found, over all supporting publishers any provider in the run found for that claim. | higher |
| Citation precision | The share of retrieved items the judge labels as evidence, supporting or contradicting. | higher |
| Duplicate inflation | Items retrieved per distinct origin: how much of the context was copies. | lower |
| Coverage reported | The share of retrieval calls whose response said whether it was complete. | higher |
| Context tokens | Evidence placed in the agent's context per claim, estimated the same way for every provider. | lower |
| Retrieval calls | Calls until the loop stopped, with a cap of four per claim. | lower |
Providers.unlob ground, unlob search, Exa, Tavily, Parallel, Linkup — each driven by the same loop and graded by the same judge, which never sees which provider a passage came from.
Results pending
No run has been published yet, so the columns above have no numbers under them. We will not type a number into this page that the harness did not produce; when the first run lands, it appears with its method and the harness itself, and you can rerun it with your own keys before believing it.
What the latency budget looks like
For an agent making several retrieval calls per turn, search latency is a real part of the user-perceived response time — and the only part you can control without changing models.
- Keyword p502.1 ms
- Semantic p503.3 ms
- Hybrid p504.6 ms
- Hybrid p996.5 ms
Frequently asked questions
What corpus were these figures measured on?
A synthetic five-vertical corpus of roughly 15,000 to 31,000 admitted passages across about 2,000 hosts, running single-node and single-thread. That is a small corpus, and we state it rather than implying the numbers were taken at web scale.
Will latency hold as the index grows?
The measured evidence is that a 2× corpus increase moved keyword p50 by 0.3 ms — from 2.1 to 2.4 — in this benchmark. That is two measurements on a small corpus: evidence of the shape of the curve, not proof at a thousand times the scale, and we will not claim otherwise until we have measured it.
Why are these serving figures only?
Because they are the only ones you can act on. Index-time rates and the resident-memory layout describe our pipeline rather than your request, and we no longer publish them — how the index is built and served is the part of this business worth protecting. Latency, throughput, filter semantics and coverage transparency all remain published in full.
What p99 should I design for?
Use the p99 column, not the median, and add your own network round trip — these are server-side figures measured at the engine. For a European caller the round trip typically dominates the query itself, which is worth knowing before you optimise the wrong half.
Do you publish query-quality benchmarks against competitors?
Not a ranking-quality shootout, and not one we ran in private. Retrieval-quality comparisons across providers are extremely sensitive to query set selection, and a benchmark we designed and ran ourselves would deserve the scepticism it got. So the agent grounding benchmark is built as a harness — the query set, the fixed agent loop, and adapters for each provider — that will be published with its first results, so anyone can rerun it with their own keys. It measures the cost of reaching a defensible answer, not search latency. The free tier is 10,000 credits a month precisely so you can run it.
Measure it yourself
The free tier is enough to run your own latency test against real queries. That is a better benchmark than ours, because it uses your workload.
Evaluating for a team or at volume? Send a workload brief or write to hello@unlob.com — a person replies.