Search is a selection problem, not a ranking problem
A human engine can return a hundred documents and rank the good one first. An agentic engine must return only the good one — because the agent cannot disambiguate the query, and cannot triage the results without paying for them.
Query
Scoped to a vertical, narrowed by filters
Match
Keyword, semantic or hybrid
Filter
On precomputed signals
Return
Metadata-only hits, ~2 ms
Fetch or traverse
One body, or the coverage graph
What happens to a request
You send a query
A query string, optionally scoped to a vertical, optionally narrowed by filters on date, language, host, content type or any of the baked signals. Scoping to a vertical is the single highest-leverage thing you can do to an ambiguous query.
It is matched three ways
Keyword mode ranks with BM25 and is right for exact identifiers. Semantic mode matches by meaning and is right for descriptive questions. Hybrid is the default: both, fused by reciprocal rank, which is the correct choice unless you know otherwise.
Signals filter and sort it
Quality, host rank, centrality, community and corroboration count are computed ahead of time and returned with every hit, so you can filter and sort on them without a second call and without paying a query-time penalty for it.
You get metadata, not bodies
A hit is a url, host, title, snippet, score and its signals. Page bodies are never included. A ten-result search costs an agent a few hundred tokens instead of tens of thousands, which is the whole point.
You fetch only what you chose
One document, full text, on a separate call — after the agent has decided which of the ten results is worth reading. Retrieval and reading are separately billed and separately controlled.
Or you traverse instead
Every result is a node in the coverage graph. related, corroborate, authorities, dossier, path and assemble_context turn twenty follow-up searches into one call, because the relationships are already computed.
Why fewer results is the feature
A person discards a bad search result in about a tenth of a second and at no cost. An agent discards one by reading it, which costs context window and reasoning tokens it will not get back. Ten mediocre results are not a neutral default for a machine reader — they are a bill.
So the useful question is not "how much of the web is in there" but "how much of what comes back is worth the tokens". We optimise for the second, accept a smaller index as the price, and make the gaps queryable rather than quietly padding results to fill a page.
Exact-match needles still work
Error codes, identifiers, citations and version strings are the case where semantic search quietly fails and a selective index could plausibly make it worse. It does not: keyword mode is a first-class path, not a fallback. The term filter.
Nothing is removed silently
Every removal is recorded with a typed reason against an append-only ledger, andwhy_not reads from it. A selective index is only trustworthy if you can interrogate what it selected against, which is why we built that first.
How we crawl
UnlobBot honours robots.txt with longest-match precedence, respects crawl-delay, identifies itself with a real user agent, and holds to roughly one request per second per host. Where a site blocks us, we accept the coverage gap and surface it through why_not rather than escalating into an arms race. It is the cheaper policy and it is the correct one.
Frequently asked questions
Why is search for agents a selection problem rather than a ranking problem?
A human search engine can return a hundred documents and rank the good one first, because a person skims and discards at a glance. An agent cannot disambiguate an ambiguous query and cannot triage results cheaply — every bad result it receives costs context and reasoning. So quality has to be decided before results are returned, not sorted afterwards.
Why is the index deliberately bounded?
Because an infinite index is a game you cannot win — you will not outspend Google at storing everything, and for an agent most of what you would store is noise that costs it context to discard. We admit selectively instead. The honest consequence is that the index does not hold everything, which is exactly why why_not exists.
What are verticals for?
Correctness, not scaling. The index is a federation of vertical cores, so a query for "rust" in the code vertical cannot return metallurgy results — those passages are not in that core. It is disambiguation by construction rather than by ranking.
How does multilingual search work without translation?
All 101 supported languages share one embedding space, so there is no translation step and no per-language index — an English query can retrieve a relevant German passage directly. Languages whose scripts do not delimit words, such as Chinese and Japanese, are tokenised as character bigrams at both index and query time.
What happens when a query finds nothing good?
You get a short honest answer rather than ten padded ones, and the gap is recorded so the crawler can close it. If you need to know why a specific URL was not there, why_not(url) will tell you: present, removed with a typed reason, or never admitted.
Why do you not publish the indexing architecture?
Because it is the part of this business that is genuinely hard to build, and publishing it helps our competitors more than it helps you. What we do publish is everything on your side of the API: the measured latency, the full filter semantics, every endpoint parameter, and a queryable answer to what is and is not in the index.
See what selective retrieval returns
The difference shows up fastest on an ambiguous query — the kind where a conventional index returns ten plausible results from four different domains.