Dense, Sparse, or Hybrid Retrieval

The situation

A team is building a retrieval-augmented support assistant over a documentation corpus: product manuals, release notes, an internal knowledge base, and a few thousand resolved support tickets. Queries come from two very different mouths. End users ask things in natural language, “why does my box arrive warm”, and internal agents paste in fragments, “error E4021”, “firmware 2.14.3”, “SKU GB-CHILL-04”. The index has to serve both.

The first cut used a pure vector store: chunk the corpus, embed every chunk, embed the query, return the nearest neighboursNearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed. . It works beautifully for the natural-language questions. Ask about warm boxes and it finds the cold-chain troubleshooting page even though that page never uses the word “warm”. Then an agent searches for “E4021” and the assistant returns three pages about unrelated cooling faults, because to the embedding model “E4021” is a low-signal token that sits near every other error-code-shaped string in the vector space. The exact match that a human would spot instantly is the exact match the vector index is worst at.

The instinct is to reach for a bigger embedding model. The actual question is whether this corpus and these queries want semantic similarity, lexical matching, or both at once, and what running both costs.

What actually matters

Dense and sparse retrieval fail in opposite directions, and the corpus decides which failure hurts more. A dense retriever embeds text into a vector and ranks by semantic closeness, so it handles synonyms, paraphrase, and “these two passages mean the same thing in different words” without anyone maintaining a thesaurus. What it gives up is precision on exact tokens. Identifiers, error strings, version numbers, part codes, and rare proper nouns carry almost no semantic signal, so the embedding for “E4021” is not meaningfully distinct from the embedding for “E4102”, and a query for one happily returns the other.

A sparse retriever ranks by term overlap, and the modern default is BM25: it scores a document by how many of the query’s terms it contains, weighted so that rare terms count for more and long documents do not win just by being long. That weighting is exactly why sparse search nails the identifiers dense search fumbles. “E4021” is a rare term, so a document containing it scores high and a document without it scores zero. The cost is the mirror image of dense’s: BM25 has no idea that “warm” and “insufficient cooling” are the same complaint, so a paraphrased query that shares no vocabulary with the answer retrieves nothing.

So the deciding property is the interaction between query vocabulary and corpus vocabulary. If users reliably say things in the words the documents use, or reliably search by exact identifiers, one retriever will do. The trouble is corpora that carry both kinds of content and take both kinds of query, which is most real support and documentation corpora, and there neither retriever alone is safe.

Hybrid retrieval runs both and combines the results, which recovers the strengths of each: the dense arm catches the paraphrase, the sparse arm catches the part number, and the fused ranking surfaces whichever arm found the better answer. The catch is that the two retrievers return scores on completely different scales, so you cannot just add them. Fusion needs the scores brought onto a comparable footing, either by normalising each retriever’s scores before a weighted blend, or by ranking-based fusion that ignores the raw scores and combines positions. That fusion step is real work, real tuning, and a second retriever to operate, so hybrid is the right default but not a free one.

The last thing worth naming is that the choice is not only about recall. Sparse indexes are cheap, interpretable (“it matched because both contained E4021”), and need no embedding model at query time; dense indexes cost embedding compute and a vector store but generalise to language the corpus author never anticipated. Hybrid pays for both.

What we’ll filter on

1. Exact-match tokens, does the corpus contain identifiers, error codes, version numbers, or rare proper nouns that users search by verbatim?
2. Query shape, are queries natural-language questions, keyword fragments, or a mix of both?
3. Vocabulary gap, do users describe things in different words from the documents (synonyms, paraphrase), or in the documents’ own words?
4. Fusion cost, is the team able to run and tune two retrievers plus a score-normalisation step?
5. Operational weight, embedding compute and a vector store versus a lexical index, and how much interpretability the answer needs.

The retrieval landscape

Dense (vector / semantic) retrieval. Embed each chunk and the query into the same vector space, rank by cosineCosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”. or dot-product similarity, return the nearest neighbours. Strong on meaning: it retrieves a passage that answers the question even when it shares no words with it, which is exactly what open-ended user questions need. Weak on the literal: exact identifiers and rare tokens blur into their neighbours, and out-of-vocabulary strings the embedding model never really learned get placed almost arbitrarily. Cost is an embedding model at index and query time plus a vector store. On AWS this is an OpenSearch k-NNk-NNThe retrieval question itself: given a query vector, return the k closest vectors under the index’s distance metric – answered exactly by comparing against everything, or quickly by an ANN index. vector field, or a Bedrock Knowledge Base backed by a vector store like OpenSearch Serverless, Aurora PostgreSQL with pgvector, or the others Bedrock supports.

Sparse (keyword / lexical, BM25) retrieval. Score documents by weighted term overlap. BM25 is the standard, tuned so rare query terms dominate and document length is normalised out. Superb on exact matches: error strings, SKUs, version numbers, function names, surnames. It is cheap, needs no embedding model, and every match is explainable by the terms that overlapped. Its blind spot is semantics: no vocabulary overlap, no match, so paraphrase and synonym queries fall through. This is a classic inverted-text index, the lexical scoring OpenSearch and Elasticsearch have always done.

Hybrid retrieval. Run a dense query and a sparse query over the same corpus and fuse the two result sets into one ranking. Because the arms cover each other’s blind spots, hybrid tends to match or beat either alone on a mixed corpus, and it degrades gracefully: on a pure-identifier query the sparse arm carries it, on a pure-paraphrase query the dense arm does. The engineering is the fusion. On Amazon OpenSearch you build a search pipeline with a normalisation processor that rescales each subquery’s scores and then combines them (arithmetic, geometric, or harmonic mean, with weights), so a hybrid query returns a single fused ranking. A Bedrock Knowledge Base exposes this more simply: over a supported vector store such as OpenSearch Serverless it offers a search-type option of SEMANTIC or HYBRID, and choosing HYBRID runs the dense and lexical retrieval and fuses them for you.

Reranking, the orthogonal lever. Not a fourth kind of retrieval, but worth flagging because it is easy to confuse with the choice. A reranker (a cross-encoderCross-encoderA model that reads a query and a passage together and scores the pair, more accurate than comparing two independently-made vectors. model) takes a candidate set that any of the above produced and re-scores each candidate against the query for relevance, cheaply improving the final ordering. It sharpens precision at the top of the list regardless of whether the candidates came from dense, sparse, or hybrid retrieval; it does not fix a candidate set that never contained the right document. Retrieval strategy decides what gets found; reranking decides how the found set is ordered.

Side by side

Property	Dense (vector)	Sparse (BM25)	Hybrid
Synonyms and paraphrase	✓	✗	✓
Exact identifiers, error codes, versions	✗	✓	✓
Rare / out-of-vocabulary tokens	✗	✓	✓
Natural-language questions	✓	✗	✓
Keyword fragment queries	✗	✓	✓
No embedding model needed	✗	✓	✗
Interpretable match reason	✗	✓	Partly
Single retriever, no fusion tuning	✓	✓	✗
Safe default for a mixed corpus	✗	✗	✓

The table reads as a coverage argument. Every row where dense fails, sparse succeeds, and vice versa; hybrid is the column with no failures except the operational ones (it needs the embedding model and the fusion step). For a corpus that is purely one shape, the matching single retriever is simpler and cheaper. The moment the corpus carries both prose and identifiers, and the queries arrive in both shapes, the single-retriever columns each have a red mark that matters and hybrid is the one that does not.

The picks in depth

For the support assistant in the situation, hybrid is the pick, and the reason is precisely the split personality of the traffic. The natural-language questions need the dense arm; the “E4021” and “firmware 2.14.3” fragments need the sparse arm; no single retriever serves both without a hole. The cleanest path is a Bedrock Knowledge Base over a supported vector store with the search type set to HYBRID, which runs both retrievals and fuses them without the team hand-building a pipeline. If the stack is OpenSearch directly rather than through Bedrock, the equivalent is a hybrid query behind a search pipeline whose normalisation processor rescales and combines the dense k-NN subquery and the BM25 subquery; the thing to tune there is the combination weights, because a corpus heavy on identifiers may want the lexical arm weighted up and a corpus heavy on prose the reverse.

Where hybrid is not the answer: a corpus with no meaningful exact-match tokens, say a collection of essays or policy prose queried in natural language, gets little from the sparse arm and can run dense alone, saving a retriever and its tuning. The mirror case is a corpus that is almost entirely identifiers and structured fragments, a parts catalogue queried by code, or logs queried by error string, where dense adds cost and noise and BM25 alone is both cheaper and more precise. Reaching for hybrid reflexively on a single-shape corpus is the same over-engineering as reaching for a bigger embedding model on the identifier problem: it spends complexity where the failure it fixes does not occur.

Two implementation notes that decide whether hybrid actually delivers. First, both arms must index the same chunks, or the fused ranking compares different populations; keep chunking and the document set identical across the dense field and the lexical field. Second, fusion is where hybrid is won or lost. Raw dense similarity scores and BM25 scores are not comparable numbers, so the normalisation step is not optional; skip it and whichever retriever happens to emit larger raw scores dominates the blend regardless of relevance. The normalisation processor exists precisely to put the two on a common scale before combining, and its weights are the knob you tune against a labelled query set.

A worked example: the two queries that broke the vector-only build

Take the corpus as described and run the two queries that exposed the problem.

Query one, from an end user: “why does my box turn up warm”. The relevant page is titled “Diagnosing insufficient cooling on delivery” and never contains the word “warm”. BM25 alone scores it near zero, because the query and the document share no content terms. The dense arm embeds the query and the page close together, because they mean the same thing, and returns it at the top. On this query the semantic arm is doing all the work.

Query two, from an internal agent: “E4021”. The relevant page is the fault reference that lists E4021 and its remedy. The dense arm places “E4021” among a cloud of similar-looking error-code tokens and returns a near-random handful of cooling-fault pages. The sparse arm treats “E4021” as a rare term, finds the one page that contains it, and scores it far above everything else. Here the lexical arm carries the query alone.

Run both through a hybrid query. Each arm returns its candidates with its own scores; the normalisation processor rescales dense similarities and BM25 scores onto a common 0-to-1 footing and combines them with the configured weights. On query one the dense contribution dominates the fused score and the cooling page wins; on query two the sparse contribution dominates and the fault reference wins. Neither query needed a human to pick which retriever to use, and neither returned the vector-only build’s wrong answers. The failure that a bigger embedding model would not have fixed is the failure the sparse arm closes for free, and the failure sparse alone would have on query one is the one the dense arm closes. That mutual cover, made usable by the normalisation step, is the whole case for hybrid.

What’s worth remembering

1. Dense and sparse retrieval fail in opposite directions: dense misses exact tokens, sparse misses paraphrase, and the corpus decides which failure hurts.
2. Dense (vector) retrieval captures meaning, so it handles synonyms and rephrasing, but identifiers, error codes, and version numbers carry little semantic signal and blur into their neighbours.
3. Sparse retrieval (BM25) scores by weighted term overlap, weighting rare terms highest, which is exactly why it nails part codes, error strings, and rare proper nouns and why it misses semantic paraphrase.
4. Hybrid runs both retrievers and fuses the results, covering each arm’s blind spot, which makes it the safe default for a corpus that carries both prose and identifiers and takes both natural-language and keyword queries.
5. The two retrievers return scores on different scales, so fusion needs score normalisation before a weighted blend; skip that step and whichever arm emits larger raw numbers dominates regardless of relevance.
6. On Amazon OpenSearch, hybrid search is a hybrid query behind a search pipeline with a normalisation processor that rescales and combines the dense and BM25 subqueries, with tunable combination weights.
7. A Bedrock Knowledge Base over a supported vector store offers a search-type option of SEMANTIC or HYBRID, running and fusing both retrievals for you without hand-building the pipeline.
8. A single-shape corpus does not need hybrid: pure prose queried in natural language can run dense alone, and a pure-identifier corpus is cheaper and more precise on BM25 alone.
9. Reranking is orthogonal to the dense-sparse-hybrid choice: it re-orders a candidate set for precision but cannot recover a document the retriever never returned.
10. Reaching for a bigger embedding model to fix an exact-match miss is the wrong lever; the miss is lexical, and the sparse arm, not a richer embedding, is what closes it.