Dense, Sparse, or Hybrid Retrieval

The situation

A team is building a retrieval-augmented support assistant over a documentation corpus: product manuals, release notes, an internal knowledge base, and a few thousand resolved support tickets. Queries arrive in two very different shapes. End users ask things in natural language, “why does my box arrive warm”, and internal agents paste in fragments, “error E4021”, “firmware 2.14.3”, “SKU GB-CHILL-04”. The index has to serve both.

The first cut used a pure vector store: chunk the corpus, embed every chunk, embed the query, return the nearest neighboursNearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed.. It works beautifully for the natural-language questions. Ask about warm boxes and it finds the cold-chain troubleshooting page even though that page never uses the word “warm”. Then an agent searches for “E4021” and the assistant returns three pages about unrelated cooling faults, because to the embedding model “E4021” is a low-signal token that sits near every other error-code-shaped string in the vector space. The exact match that a human would spot instantly is the exact match the vector index is worst at.

The instinct is to reach for a bigger embedding model. The actual question is whether this corpus and these queries need semantic similarity, lexical matching, or both at once, and what running both costs.

What actually matters

Dense and sparse retrieval fail in opposite directions, and which failure hurts more depends on the corpus. A dense retriever embeds text into a vector and ranks by semantic closeness, so it handles synonyms, paraphrase, and “these two passages mean the same thing in different words” without anyone maintaining a thesaurus. What it gives up is precision on exact tokens. Identifiers, error strings, version numbers, part codes, and rare proper nouns carry almost no semantic signal, so the embedding for “E4021” is not meaningfully distinct from the embedding for “E4102”, and a query for one returns the other.

A sparse retriever ranks by term overlap, and the modern default is BM25: it scores a document by how many of the query’s terms it contains, weighted so that rare terms count for more and long documents do not score highly just for being long. That weighting is why sparse search matches the identifiers dense search misses. “E4021” is a rare term, so a document containing it scores high and a document without it scores zero. The weakness is the mirror image of dense’s. BM25 scores only on shared terms, so a query saying “warm” never reaches a page that says “insufficient cooling”, and a paraphrase that shares no vocabulary with the answer retrieves nothing.

So the deciding property is the interaction between query vocabulary and corpus vocabulary. If users reliably say things in the words the documents use, or reliably search by exact identifiers, one retriever will do. The trouble is corpora that carry both kinds of content and take both kinds of query, which is most real support and documentation corpora, and there neither retriever alone is safe.

Hybrid retrieval runs both and combines the results, which recovers the strengths of each: the dense arm catches the paraphrase, the sparse arm catches the part number, and the fused ranking surfaces whichever arm found the better answer. The catch is that the two retrievers return scores on completely different scales, so you cannot just add them. Fusion needs each retriever’s scores rescaled onto a comparable footing first, then combined in a weighted blend. That step is real work and real tuning on top of a second retriever to operate, so hybrid is the right default and the heavier one to build.

The choice is not only about recall. A lexical index needs no embedding model at query time and every match is explainable by the overlapping terms; a dense index needs embedding compute and a vector store, and generalises to language the corpus author never anticipated. Hybrid runs both.

What we’ll filter on

1. Exact-match tokens, does the corpus contain identifiers, error codes, version numbers, or rare proper nouns that users search by verbatim?
2. Query shape, are queries natural-language questions, keyword fragments, or a mix of both?
3. Vocabulary gap, do users describe things in different words from the documents (synonyms, paraphrase), or in the documents’ own words?
4. Fusion cost, is the team able to run and tune two retrievers plus a score-normalisation step?
5. Operational weight, embedding compute and a vector store versus a lexical index, and how much interpretability the answer needs.

The landscape

Dense (vector / semantic) retrieval. Embed each chunk and the query into the same vector space, rank by cosineCosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”. or dot-product similarity, return the nearest neighbours. Strong on meaning: it retrieves a passage that answers the question even when it shares no words with it, which is exactly what open-ended user questions need. Weak on the literal: exact identifiers and rare tokens blur into their neighbours, and out-of-vocabulary strings absent from the model’s training data land almost arbitrarily. Cost is an embedding model at index and query time plus a vector store. On AWS this is an OpenSearch k-NNk-NNThe retrieval question itself: given a query vector, return the k closest vectors under the index’s distance metric – answered exactly by comparing against everything, or quickly by an ANN index. vector field, or a Bedrock Knowledge Base backed by one of its supported vector stores: OpenSearch Serverless, an OpenSearch managed cluster, Aurora PostgreSQL with pgvector, S3 Vectors, Neptune Analytics, Pinecone, MongoDB Atlas, or Redis Enterprise Cloud.

Sparse (keyword / lexical, BM25) retrieval. Score documents by weighted term overlap. Okapi BM25 is the standard, and it is what OpenSearch scores a keyword query with by default: rare query terms dominate and document length is normalised out. Superb on exact matches: error strings, SKUs, version numbers, function names, surnames. It needs no embedding model, and every match is explainable by the terms that overlapped. Its blind spot is semantics: no vocabulary overlap, no match, so paraphrase and synonym queries fall through. This is a classic inverted-text index, the lexical scoring OpenSearch and Elasticsearch have always done.

Hybrid retrieval. Run a dense query and a sparse query over the same corpus and fuse the two result sets into one ranking. Because the arms cover each other’s blind spots, hybrid degrades gracefully: on a pure-identifier query the sparse arm carries it, on a pure-paraphrase query the dense arm does. The engineering is the fusion. On Amazon OpenSearch you send a hybrid query and attach a search pipeline containing a normalisation processor. It rescales each subquery’s scores with min_max or l2, then combines them with an arithmetic_mean, geometric_mean or harmonic_mean, weighted per subquery, and one fused ranking comes back. A Bedrock Knowledge Base exposes the same idea as a setting, overrideSearchType of SEMANTIC or HYBRID on a Retrieve or RetrieveAndGenerate call. HYBRID applies only to Amazon RDS, OpenSearch Serverless and MongoDB Atlas vector stores that contain a filterable text field. On any other store, or one without that field, the query runs semantic search instead.

Reranking, the orthogonal lever. Not a fourth kind of retrieval, but worth flagging because it is easy to confuse with the choice. A reranker model takes a candidate set that any of the above produced, scores each candidate’s relevance to the query, and reorders the set by those scores. Amazon Bedrock offers this as the Rerank API operation and as a reranking configuration on a Knowledge Base Retrieve or RetrieveAndGenerate call, for text data only. It sharpens precision at the top of the list regardless of whether the candidates came from dense, sparse, or hybrid retrieval; it does not fix a candidate set that never contained the right document. Retrieval strategy sets what gets found; reranking sets the order of what was found.

Evaluation

Side by side

Property	Dense (vector)	Sparse (BM25)	Hybrid
Synonyms and paraphrase	✓	✗	✓
Exact identifiers, error codes, versions	✗	✓	✓
Rare / out-of-vocabulary tokens	✗	✓	✓
Natural-language questions	✓	✗	✓
Keyword fragment queries	✗	✓	✓
No embedding model needed	✗	✓	✗
Interpretable match reason	✗	✓	Partly
Single retriever, no fusion tuning	✓	✓	✗
Safe default for a mixed corpus	✗	✗	✓

The table reads as a coverage argument. Every row where dense fails, sparse succeeds, and vice versa; hybrid is the column with no failures except the operational ones (it needs the embedding model and the fusion step). For a corpus that is purely one shape, the matching single retriever is simpler and cheaper. The moment the corpus carries both prose and identifiers, and the queries arrive in both shapes, the single-retriever columns each carry a ✗ that matters while hybrid does not.

The solution

For the support assistant in the situation, hybrid is the pick, and the reason is precisely the two shapes of the traffic. The natural-language questions need the dense arm; the “E4021” and “firmware 2.14.3” fragments need the sparse arm; no single retriever serves both without a hole. The cleanest path is a Bedrock Knowledge Base with the search type set to HYBRID, which runs both retrievals and fuses them without the team hand-building a pipeline. Check the vector store before committing to that, because only Aurora (RDS), OpenSearch Serverless and MongoDB Atlas support the setting; on S3 Vectors or Pinecone the same request retrieves semantically and the identifier problem survives. If the stack is OpenSearch directly rather than through Bedrock, the equivalent is a hybrid query behind a search pipeline whose normalisation processor rescales and combines the dense k-NN subquery and the BM25 subquery; the thing to tune there is the combination weights, because a corpus heavy on identifiers may call for the lexical arm weighted up and a corpus heavy on prose the reverse.

Where hybrid is not the answer: a corpus with no meaningful exact-match tokens, say a collection of essays or policy prose queried in natural language, gets little from the sparse arm and can run dense alone, saving a retriever and its tuning. The mirror case is a corpus that is almost entirely identifiers and structured fragments, a parts catalogue queried by code, or logs queried by error string, where dense adds cost and noise and BM25 alone is both cheaper and more precise. Reaching for hybrid reflexively on a single-shape corpus is the same over-engineering as reaching for a bigger embedding model on the identifier problem: it adds complexity where the failure it fixes does not occur.

Two implementation notes that decide whether hybrid actually delivers. First, both arms must index the same chunks, or the fused ranking compares different populations; keep chunking and the document set identical across the dense field and the lexical field. Second, raw dense similarity scores and BM25 scores are not comparable numbers, so the normalisation step is not optional. Skip it and whichever retriever emits larger raw scores dominates the blend regardless of relevance. The normalisation processor puts the two on a common scale before combining, and its weights are the knob you tune against a labelled query set.

Worked example

Take the corpus as described and run the two queries that exposed the problem.

Query one, from an end user: “why does my box turn up warm”. The relevant page is titled “Diagnosing insufficient cooling on delivery” and never contains the word “warm”. BM25 alone scores it near zero, because the query and the document share no content terms. The dense arm embeds the query and the page close together, because they mean the same thing, and returns it at the top. On this query the semantic arm is doing all the work.

Query two, from an internal agent: “E4021”. The relevant page is the fault reference that lists E4021 and its remedy. The dense arm places “E4021” among a cloud of similar-looking error-code tokens and returns a near-random handful of cooling-fault pages. The sparse arm treats “E4021” as a rare term, finds the one page that contains it, and scores it far above everything else. Here the lexical arm carries the query alone.

Run both through a hybrid query. Each arm returns its candidates with its own scores; the normalisation processor rescales dense similarities and BM25 scores onto a common 0-to-1 footing and combines them with the configured weights. On query one the dense contribution dominates the fused score and the cooling page wins; on query two the sparse contribution dominates and the fault reference wins. Neither query needed a human to pick which retriever to use, and neither returned the vector-only build’s wrong answers. A bigger embedding model would not have rescued query two; the sparse arm does. Sparse alone would have failed query one; the dense arm covers it.

What’s worth remembering

1. Dense and sparse retrieval fail in opposite directions: dense misses exact tokens, sparse misses paraphrase, and which failure hurts depends on the corpus.
2. Hybrid runs both retrievers and fuses the results, covering each arm’s blind spot. That makes it the safe default for a corpus carrying both prose and identifiers, taking both natural-language and keyword queries.
3. The two retrievers return scores on different scales, so fusion needs score normalisation before a weighted blend; skip that step and whichever arm emits larger raw numbers dominates regardless of relevance.
4. A single-shape corpus does not need hybrid: pure prose queried in natural language can run dense alone, and a pure-identifier corpus is cheaper and more precise on BM25 alone.
5. Reaching for a bigger embedding model to fix an exact-match miss is the wrong lever; the miss is lexical, and the sparse arm closes it.
6. Bedrock’s HYBRID search type applies only to Aurora (RDS), OpenSearch Serverless and MongoDB Atlas vector stores that contain a filterable text field; on any other store the query runs semantic search.