Use case
RAG over the open web
Teams grounding an LLM in current web content rather than a private corpus.
The problem, and the approach
The problem
- A RAG pipeline over the web is not the same problem as RAG over your own documents. The corpus is adversarial, enormously redundant, and of wildly varying quality — three properties your internal wiki does not have.
- Most of the resulting code is not retrieval. It is deduplication, trust assessment, reranking and truncation, written once and rarely revisited, and it is where answer quality is actually decided.
The approach
- Search returns metadata with trust signals attached, so the selection step happens before any page text is fetched or paid for.
- Story collapsing removes syndicated duplicates at the source rather than in your code, and corroboration counts survive that deduplication.
- assemble_context runs the entire loop server-side and returns a packed set with a reason per passage — usually replacing the pipeline outright.
The parameters that matter
| Parameter | Value | Why |
|---|---|---|
collapse | story | One representative per near-duplicate cluster, so ten slots carry ten facts. |
min_independent_sources | 2 | Single-sourced claims never enter the context. |
min_quality | 60 | Removes thin and boilerplate-heavy pages before they cost tokens. |
fields[] | id, url, title, snippet, independent_sources | Everything the selection step reads, and nothing else. |
# The whole loop, server-side
curl -H "x-api-key: $UNLOB_API_KEY" \
"https://api.unlob.com/assemble_context?q=current+state+of+EU+AI+regulation&budget=4000"Where web RAG stops resembling document RAG
The playbook for RAG over a private corpus assumes a cooperative collection: you own it, it is roughly deduplicated, the authorship is known, and nothing in it is trying to rank. Every one of those assumptions fails on the open web, and the pipeline that worked over an internal wiki degrades in a way that is hard to see, because it still returns ten plausible passages.
The redundancy is the first thing to bite. A single announcement can occupy an entire result set as syndicated copies, and a naive top-k then packs one fact ten times into a context window that could have held ten facts. The fix is structural rather than a reranking tweak: collapse near-duplicates before selection, and keep the count of how many distinct hosts carried the claim so the model still knows the difference between widely reported and merely widely copied.
Budget packing is a ranking decision in disguise
Truncating to a token budget is usually written as the last step of the pipeline and treated as arithmetic. It is not. Whatever falls off the end of the list is the passage the model will never see, so the truncation rule is a ranking policy that nobody reviewed and nobody tests.
Making it explicit is cheap. Decide whether the budget should be spent on breadth (one passage each from several independent sources) or on depth (several passages from the best source), and say which — the two produce visibly different answers on contested subjects. assemble_context takes the budget as a parameter and returns a reason per included passage, which at least makes the policy inspectable; if you keep your own pipeline, log what was dropped, because that log is where the answer quality regressions show up first.
Where this is not the right tool
- The index is bounded and smaller than Google or the well-funded competitors. For broad consumer topics you may find gaps — why_not will tell you where.
- assemble_context uses our ranking policy. If you have a reranker you trust, use the retrieval endpoints and keep your own pipeline.
Frequently asked questions
Should I use assemble_context or build my own pipeline?
assemble_context when you would rather not maintain deduplication, trust ranking and budget packing — it does all three and explains each inclusion. Your own pipeline when you have a reranker or domain-specific ranking you trust more than a general policy.
How do I handle content the index does not have?
Call why_not on the URL you expected. It distinguishes never-crawled from removed-and-why, which tells you whether to supplement with your own extraction or accept the gap.
Build it on the free tier
10,000 credits a month, no card. Every parameter above works on every plan.