Skip to content
unlob
Get an API key

Ask any search index why a URL is missing. Only one will answer.

Search APIs are precise about what they found and silent about what they did not. That silence is where agents hallucinate and where audits fail. We keep an append-only ledger of every removal decision, and we let you query it.

Coverage transparencybash
curl -H "x-api-key: $UNLOB_API_KEY" \
  "https://api.unlob.com/why_not?url=https://example.com/docs/v1/auth"

{
  "status": "removed",
  "reason": "superseded",
  "superseded_by": "p:41ab7c",
  "removed_at": 1751328000
}

Three facts that look identical

An empty result set from any other search API could mean any of these. Your agent has to pick one, and it has nothing to pick with.

It does not exist

No such page was ever published. The correct answer is "there is nothing to find".

It exists, we never saw it

Outside the crawl, blocked by robots, or in a region of the web we have not reached. The correct answer is "we do not know".

We had it and removed it

Superseded, deduplicated, aged out or displaced. The correct answer names the reason — and often the replacement.

The six removal reasons

The index is deliberately bounded — it admits a curated core and covers the tail on demand, rather than growing without limit. That means passages leave, and every departure is recorded with a reason.

redundantA near-duplicate of a passage already indexed.

The surviving copy is named in the response. The removed copy’s host is still recorded against it, so corroboration counts are unaffected.

supersededA newer version of the same page replaced it.

The response carries superseded_by, pointing at the replacement passage.

staleThe content aged out of the retention window for its class.

Different content types have different useful lifetimes; documentation outlives a news wire.

access-starvedNothing has retrieved it for long enough to justify the space.

The index is bounded, so a passage nothing ever retrieves eventually gives up its place to one that is being asked for.

lost-replacementA stronger passage covering the same ground took its place.

The exception is exact-match: an identifier that would become unfindable is never displaced this way, which is what makes the term filter dependable.

tombstoned-sourceThe source asked to be removed, or became unfetchable.

Robots changes, takedowns and permanent errors all land here.

One thing is never removed

No pruning or replacement may remove the last indexed carrier of a salient term. An error code, a CVE identifier, a part number or a case citation stays findable no matter how aggressively the surrounding corpus is compacted — because an exact-match needle has no semantic neighbourhood to fall back on. Lose it and it is simply gone.

Provenance on every result

Transparency about absence is only half of it. Every hit that does come back carries how it got into the index and how well it is corroborated.

source

cc for the batch corpus, delta for a page the crawler fetched for freshness or gap-fill.

fetched_at

When we retrieved it — the audit clock, distinct from publication.

published_at

When the content says it was published — the recency you usually mean.

independent_sources

How many distinct hosts assert this story, counted across rejected duplicates too.

If you have to defend a retrieval decision

In finance, legal, pharma and the public sector, the question after an agent produces an answer is rarely "what did it find". It is "what did it miss, and can you show that it was not material". No amount of ranking quality answers that. A queryable record of what was excluded and why does.

  • Reconstruct the evidence base at a point in time with from and to on crawl timestamps.
  • Demonstrate corroboration thresholds with min_independent_sources.
  • Explain any specific exclusion with why_not.
  • Show the reason each included passage was chosen, from assemble_context.

Frequently asked questions

What does why_not actually return?

One of three answers. "present" means the URL is in the index. "removed" carries a typed reason — redundant, superseded, stale, access-starved, lost-replacement or tombstoned-source — plus the replacement passage where one exists and the timestamp of removal. "unknown" means it was never admitted.

Why does absence matter?

Because an empty result set is ambiguous, and agents resolve ambiguity badly. "Not in the index", "removed last week", and "does not exist on the web" are three very different facts, and an agent that cannot distinguish them will confidently report the wrong one.

Do other search APIs offer this?

Not that we are aware of. You can ask Exa, Parallel, Brave or any SERP reseller for a URL and get nothing back, but none of them will tell you whether the page was rejected, dropped, or never crawled. It is a genuine gap in the category rather than a feature everyone has.

Why does this matter for regulated industries?

Because "we searched the web" is not an auditable statement and "here is what was considered, and here is what was excluded and why" is. For finance, legal, pharma and public-sector work, the ability to reconstruct why a document was or was not in scope is frequently a procurement requirement rather than a nice-to-have.

Can a passage be removed and then come back?

Yes. Removal is a bounded-index decision, not a permanent verdict. If demand returns for a region of the corpus that was pruned, the delta crawler refills it and the passage is re-admitted. The ledger records both events.

Is the removal ledger complete?

It is append-only and covers every removal decision the index makes. It does not cover pages we never crawled — for those, why_not answers "unknown", which is the honest response to "we have not seen this".

Check a URL you care about

Start free and run why_not against a page you expect to find. It is the fastest way to understand what a bounded, audited index actually contains.

Get your API keywhy_not reference