Web search for AI agents

The inversion: precision beats recall

Consumer search is a recall-and-rank problem. Return everything plausibly relevant, order it well, and let the reader discard the rest at a glance. Discarding is free for a person, so a hundred results with the good one first is a fine outcome.

For an agent, discarding is not free. Every result occupies context and every judgement about it consumes reasoning tokens. Ten results where three are junk is not "seventy percent good" — it is seventy percent good at 100% of the cost, plus whatever the agent does with the junk before rejecting it.

This is why filtering aggressively usually improves agent output. Most teams under-filter because under-filtering feels safe: a narrower query might miss something. It might, and the alternative is paying for noise on every single call.

Return metadata, fetch text on demand

A search that returns ten full page bodies has spent the context window before the agent decided which page it wanted. A search that returns ten titles, snippets and trust signals costs a fraction, and the agent then fetches the one or two passages worth reading in full.

This is why unlob search is metadata-only and /doc/:id is a separate call. It is one extra round trip for the passages you actually want, against a large saving on the ones you do not.

# 1. Cheap: metadata for ten candidates
curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/search?q=vector+index+memory&limit=10&fields[]=id&fields[]=url&fields[]=title&fields[]=snippet"

# 2. Only for what you selected
curl -H "x-api-key: $UNLOB_API_KEY" "https://api.unlob.com/doc/p:8f2c9a"

Trust signals an agent can act on

A person glances at a domain name and forms a judgement instantly. An agent has no equivalent intuition, and prompting it to "prefer authoritative sources" pushes the problem into the model rather than solving it.

Structured signals do solve it. Host rank scores the publisher, quality scores the page, centrality scores how load-bearing a passage is in its field, and independent source counts say how many distinct outlets carry the claim. All four are filterable, so trust becomes a query parameter rather than a prompt instruction.

The most valuable is usually corroboration. Set min_independent_sources=3 and single-sourced claims never reach the agent at all.

curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/search?q=merger+announced&min_independent_sources=3&min_host_rank=0.5&collapse=story"

Make absence explainable

An empty result set is ambiguous. It might mean the page does not exist, was never crawled, or was indexed and later removed. Those are three different facts and an agent asked to resolve them will guess, usually confidently.

This is a common and under-discussed source of hallucination: the model infers non-existence from an absent result, and reports it. Giving the agent a tool that answers the question directly removes the inference entirely.

curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/why_not?url=https://example.com/docs/v1/auth"
# -> present | removed (with reason) | unknown

Stop making the agent rebuild the world

The expensive part of agentic retrieval is rarely the search call. It is what happens afterwards: deduplicating results, working out which sources are independent, judging which are authoritative, deciding what to read next. That reasoning happens in the context window and is billed by the token, and it is reconstructing structure the search engine already computed.

If the index knows which passages belong to the same story, which hosts are authoritative, and which entities co-occur, it can answer those questions directly — one call instead of twenty.