Web search and retrieval glossary
63 terms, defined as the field understands them. Where we do something specific, it is marked as ours rather than presented as the general case.
Verified 5 August 2026
Retrieval
15 termsHow a query becomes a set of results.
Hybrid search
Hybrid search runs a lexical query and a vector query over the same corpus and fuses the two rankings into one result set.
BM25
BM25 is a ranking function that scores documents by term frequency and inverse document frequency, with saturation so repeated terms have diminishing returns.
Semantic search
Semantic search retrieves documents by meaning rather than by word overlap, using vector embeddings to place queries and documents in a shared space.
Vector search
Vector search finds the nearest neighbours of a query vector among document vectors, usually using an approximate index because exact search over millions of vectors is too slow.
Approximate nearest neighbour (ANN)
ANN search finds vectors close to a query vector without guaranteeing the exact closest, trading a small recall loss for orders-of-magnitude speed improvement.
Reciprocal rank fusion
Reciprocal rank fusion merges several ranked lists by scoring each document by the reciprocal of its position in each list, requiring no score normalisation.
Embedding
An embedding is a fixed-length vector representing the meaning of a piece of text, produced by a model so that similar texts land near each other.
Quantisation
Quantisation compresses vectors by storing each dimension in fewer bits, cutting memory and speeding comparison at a small cost in precision.
Reranking
Reranking reorders an initial candidate set with a more expensive, more accurate model than the one used to retrieve it.
Query expansion
Query expansion adds related terms to a query to improve recall when the user’s vocabulary differs from the documents’.
Recall and precision
Recall is the share of relevant documents a search returns; precision is the share of returned documents that are relevant.
Ranking
Ranking orders retrieved documents by predicted relevance, traditionally combining a text-matching score with query-independent quality signals.
Faceted search
Faceted search returns counts across categorical dimensions alongside results, letting a user or agent see the shape of a result set before reading it.
Filtering
Filtering restricts a search to documents matching structured criteria — date, language, domain, quality — before or during scoring.
Result collapsing
Collapsing groups near-identical results and returns one representative each, so a single syndicated item cannot occupy an entire result set.
Indexing
12 termsWhat goes into a search index, and what it costs to keep there.
Admission control
Admission control decides at index time which documents are worth indexing at all, rather than indexing everything and sorting it out at query time.
Salient term
A salient term is a distinctive, low-frequency token that identifies specific content — an error code, an identifier, a proper noun — as opposed to common vocabulary.
Bounded index
A bounded index caps its own size and prunes or replaces existing entries rather than growing without limit.
Index pruning
Pruning removes documents from an index to reclaim space, based on redundancy, staleness, access patterns or competitive replacement.
Removal ledger
A removal ledger is an append-only record of every document removed from an index, with the reason and any replacement.
Coverage transparency
Coverage transparency is the ability to ask a search index why a specific URL is or is not present, and get a substantive answer.
Inverted index
An inverted index maps each term to the list of documents containing it, which is what makes lexical search fast.
Fast field
A fast field is a document attribute stored column-wise in the index for direct lookup, enabling filtering and sorting without reading the document.
Resident memory
Resident memory is the portion of an index that must stay in RAM to serve queries, and it is usually what caps how large an index can economically grow.
Sharding
Sharding splits an index across several nodes, each holding a portion, with a gateway scattering queries and gathering results.
Near-duplicate detection
Near-duplicate detection identifies documents with substantially the same content despite differing in wording, formatting or boilerplate.
Deduplication
Deduplication removes redundant copies of the same content from an index or a result set.
Graphs & trust
11 termsRelationships between documents, and judging what to believe.
Coverage graph
A coverage graph is a typed graph over a search index recording how documents relate — who published them, which report the same story, what they are about and who links to whom.
GraphRAG
GraphRAG is retrieval-augmented generation over a knowledge graph, retrieving connected subgraphs of entities and relationships rather than isolated text chunks.
Knowledge graph
A knowledge graph stores entities and the relationships between them as a queryable structure of typed nodes and edges.
Centrality
Centrality measures how well-connected a node is within a graph, and is used to identify the documents a field treats as foundational.
Host rank
Host rank scores the authority of a publishing domain from the link graph, independent of any particular query.
Corroboration
Corroboration counts how many independent sources assert the same claim, distinguishing widely reported facts from one story copied many times.
Story cluster
A story cluster groups documents reporting the same underlying event, across publishers and wordings.
Entity extraction
Entity extraction identifies people, organisations, places and products mentioned in text and links them to canonical identifiers.
Entity dossier
A dossier is a one-hop summary of an entity: where it is mentioned, which sources cover it, and which entities appear alongside it.
Community detection
Community detection partitions a graph into clusters of nodes that reference each other far more than the rest of the graph.
Provenance
Provenance records where a document came from and how it entered the index — which crawl, when fetched, and when published.
Agents & context
10 termsHow autonomous systems consume retrieval.
Agentic search
Agentic search is web search consumed by an autonomous system rather than a person, which inverts most of the design assumptions of consumer search.
Retrieval-augmented generation (RAG)
RAG retrieves relevant documents and places them in a model’s context so it can answer from current, specific sources rather than from training data alone.
Context assembly
Context assembly selects, orders and truncates retrieved passages to fit a token budget before they enter a model’s context.
Context window
The context window is the maximum amount of text a model can consider at once, and it is the budget every retrieval decision spends against.
Model Context Protocol (MCP)
MCP is an open protocol for exposing tools and data sources to AI models through a uniform interface, so any compatible client can use any compatible server.
Tool use
Tool use lets a model invoke external functions — search, calculation, database queries — and incorporate the results into its reasoning.
Function calling
Function calling is a model’s ability to emit structured arguments for a named function rather than free text, enabling reliable tool invocation.
Hallucination
A hallucination is a confident, plausible model output that is not supported by any source.
Chunking
Chunking splits documents into passages small enough to embed and retrieve independently.
Passage
A passage is a retrievable unit of text — typically a section or several paragraphs — rather than a whole document.
Crawling
7 termsGetting the web in the first place.
Web crawler
A web crawler discovers and fetches pages by following links, respecting robots.txt and rate limits.
robots.txt
robots.txt is a file at a site’s root telling crawlers which paths they may fetch and how fast.
Crawl frontier
The crawl frontier is the prioritised queue of URLs a crawler intends to fetch, subject to per-host politeness constraints.
Crawl-on-miss
Crawl-on-miss triggers a fetch when a query finds nothing adequate, so the index fills gaps where there is demonstrated demand.
Freshness
Freshness is how current an index is relative to the live web, maintained by recrawling changed pages rather than everything.
Boilerplate removal
Boilerplate removal strips navigation, headers, footers, cookie banners and adverts from a page to leave the actual content.
Content extraction
Content extraction converts a fetched page into clean structured text, handling HTML, JavaScript-rendered pages and documents.
Operations
8 termsRunning and paying for all of it.
Object-storage-first architecture
An object-storage-first architecture keeps the authoritative copy of the data in object storage and treats compute nodes as replaceable caches rather than as the system of record.
Stateless serving
A stateless serving tier holds no authoritative data, so any node can be replaced without coordination or data loss.
Quality score
A quality score rates an individual passage on text density, structure and extraction confidence, independently of its publisher’s authority.
Authority
Authority is a coarse classification of a domain’s institutional class — educational, governmental, organisational, commercial — derived from the domain itself.
Multilingual search
Multilingual search retrieves documents across languages, ideally by placing all languages in one embedding space so no translation step is needed.
Tokenisation
Tokenisation splits text into indexable units — usually words, but character n-grams for scripts without word delimiters.
Rate limiting
Rate limiting caps how many requests a client may make per interval, protecting a service from overload and enforcing plan tiers.
Quota
A quota is the total number of requests allowed in a billing period, as distinct from a rate limit on requests per minute.
From definitions to a working index
Everything defined here is implemented in a search API you can call today. 10,000 free requests a month, no card.