RAG over the open web
Teams grounding an LLM in current web content rather than a private corpus.
The problem
- A RAG pipeline over the web is not the same problem as RAG over your own documents. The corpus is adversarial, enormously redundant, and of wildly varying quality — three properties your internal wiki does not have.
- Most of the resulting code is not retrieval. It is deduplication, trust assessment, reranking and truncation, written once and rarely revisited, and it is where answer quality is actually decided.
The approach
- Search returns metadata with trust signals attached, so the selection step happens before any page text is fetched or paid for.
- Story collapsing removes syndicated duplicates at the source rather than in your code, and corroboration counts survive that deduplication.
- assemble_context runs the entire loop server-side and returns a packed set with a reason per passage — usually replacing the pipeline outright.
The parameters that matter
| Parameter | Value | Why |
|---|---|---|
collapse | story | One representative per near-duplicate cluster, so ten slots carry ten facts. |
min_independent_sources | 2 | Single-sourced claims never enter the context. |
min_quality | 60 | Removes thin and boilerplate-heavy pages before they cost tokens. |
fields[] | id, url, title, snippet, independent_sources | Everything the selection step reads, and nothing else. |
# The whole loop, server-side
curl -H "x-api-key: $UNLOB_API_KEY" \
"https://api.unlob.com/assemble_context?q=current+state+of+EU+AI+regulation&budget=4000"Where this is not the right tool
- The index is bounded and smaller than Google or the well-funded competitors. For broad consumer topics you may find gaps — why_not will tell you where.
- assemble_context uses our ranking policy. If you have a reranker you trust, use the retrieval endpoints and keep your own pipeline.
Frequently asked questions
Should I use assemble_context or build my own pipeline?
assemble_context when you would rather not maintain deduplication, trust ranking and budget packing — it does all three and explains each inclusion. Your own pipeline when you have a reranker or domain-specific ranking you trust more than a general policy.
How do I handle content the index does not have?
Call why_not on the URL you expected. It distinguishes never-crawled from removed-and-why, which tells you whether to supplement with your own extraction or accept the gap.
Build it on the free tier
10,000 requests a month, no card. Every parameter above works on every plan.