Hybrid Search and Reranking for Bedrock RAG

The situation

The internal support assistant answers questions over product manuals, firmware release notes, and a decade of resolved tickets. It runs a Bedrock Knowledge Base with pure semantic retrieval: embed the query, pull the top five chunks by vectorVectorAn ordered list of numbers – in AI usage, almost always an embedding – and by extension the databases that index them for nearest-neighbour search. similarity, hand them to the modelModelA trained set of weights plus the architecture that makes them useful – the thing you load up and run inference against. . On conceptual questions (“how do I reset the thermostat schedule?”) it works well. Paraphrase is its strength, and the embedding model earns its keep.

The complaints are all the same shape. A field engineer types “ERR-4021 on firmware 2.3” and gets back three chunks about other error codes, a general troubleshooting overview, and one paragraph that mentions firmware 2.x in passing. The one release note that documents ERR-4021 specifically is sitting at rank 14, outside the window that ever reaches the model. The answer the assistant generates is confident, groundedGroundingConstraining a model to answer from provided sources rather than from whatever it absorbed during training. in the wrong chunks, and wrong.

The pattern is exact-term queries. Product codes, error codes, part numbers, acronyms, proper names. The tokens that carry the whole meaning of the query are precisely the tokens dense retrieval smears together. Retrieval precision on that slice of traffic needs to come up.

What actually matters

A dense retriever embeds the query and each chunk into the same space with a bi-encoder, then compares the two vectors. That comparison is why paraphrase works: “reset the schedule” and “clear the programmed times” land near each other even with no shared words. It is also why exact terms fail. ERR-4021 has almost no semantic content of its own; its embedding is dominated by the pattern “an error code,” so it sits in a tight cluster with ERR-4020, ERR-4102, and every other code the corpus has ever seen. The vectors that should be far apart are close, and similarity ranking can’t tell them apart.

Sparse retrieval is the opposite instrument. BM25 scores documents by exact token overlap, weighted by how rare each token is across the corpus. A rare token like ERR-4021 gets a high weight the moment it appears, so the one document containing it shoots to the top. The cost is that BM25 has no idea “reset the schedule” and “clear the programmed times” are the same request. It matches strings, not meaning.

Hybrid search runs both and fuses the scores. The exact-term query gets BM25’s precision on the rare token; the paraphrase query gets the embedding model’s semantic reach; a mixed query gets a blend. Fusion is where the tuning lives, normalising two score distributions that aren’t on the same scale and weighting their contributions.

Fusion lifts the right document into contention, but it doesn’t guarantee rank one. That’s the reranker’s job. A first-stage retriever, dense or sparse, scores every candidate independently: it embeds the query once, embeds each document once, and compares. A cross-encoder reranker instead reads the query and one candidate document together in a single pass and scores their actual relevance. Attending to both at once, it catches relevance signals the two independent vectors never encode. It is far more precise than the first-stage score and far more expensive, so it only ever runs over a shortlist.

That fixes the lever order. Chunking decides what a document even is; retrieval method (dense, sparse, hybrid) decides what makes the shortlist; the reranker reorders the shortlist by true relevance; the top of the reordered list goes to the model. Retrieve wide, rerank narrow: pull a generous top-N so the right document is somewhere in the candidates, then let the reranker promote it into the small top-k the context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined. can afford.

What we’ll filter on

1. Exact-term queries, does the method surface rare tokens (codes, part numbers, proper names)?
2. Paraphrase, does it still handle semantically-similar-but-differently-worded queries?
3. Final precision, how good is the small top-k that actually reaches the model?
4. Added latency per query, what does the method cost the p99 retrieval budget?
5. Added cost per query, extra model or index calls per request?
6. Managed availability, is it a first-class Bedrock or OpenSearch feature or bespoke plumbing?

The retrieval landscape

1. Dense / vector-only search. The baseline the assistant already runs. A bi-encoder embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model maps query and chunks into one space; retrieval is approximate-nearest-neighbour over the chunk vectors. Strong recall on paraphrase and conceptual questions, weak on exact tokens. No extra latency beyond the one ANN lookup, no extra cost beyond the query embedding. It is the thing to improve, not the answer.

2. Sparse / keyword (BM25). Classic lexical retrieval, scoring by rare-token overlap. Nails exact terms, misses paraphrase entirely. Available as a plain OpenSearch match query. On its own it trades one failure mode for the opposite one, so it’s a component, not a destination.

3. Hybrid search. Run dense and sparse together and fuse. Amazon OpenSearch supports this directly: a hybrid query with a search pipeline whose normalization-processor normalises the BM25 and k-NN score distributions and combines them with configurable weights, in one round trip. Amazon Bedrock Knowledge Bases exposes the same idea more simply, an overrideSearchType on the retrieve step set to SEMANTIC or HYBRID. Hybrid is the direct answer to a corpus with mixed query styles, and it adds essentially no latency over dense alone.

4. Reranking. A second stage, not a retriever. The Amazon Bedrock Rerank API takes the query and a list of retrieved documents and returns them reordered by relevance, using a cross-encoder reranker model (Cohere Rerank, Amazon Rerank). Bedrock Knowledge Bases can apply a reranker inside the retrieve step, reordering the retrieved chunks before generation. This is the biggest precision lever available, and the most expensive per query, because it’s another model call over N documents.

5. Query decomposition. Bedrock Knowledge Bases can split a multi-part question (“compare ERR-4021 and ERR-4102 behaviour on firmware 2.3”) into sub-queries, retrieve for each, and merge. It raises recall on compound questions rather than precision on a single term, so it’s complementary, not a substitute.

6. Metadata filtering. An orthogonal precision lever: restrict candidates by structured attributes (product line, firmware version, document type) before or after the vector match. It narrows the field cheaply when the query carries a hard constraint, and it stacks with any of the above.

Side by side

No single row is the whole answer. Hybrid fixes what dense misses on exact terms; the reranker fixes what any first-stage ranking leaves in the wrong order. For this corpus the two stack: hybrid gets ERR-4021 into the candidate set, the reranker gets it to rank one.

The retrieval pipeline

The pick in depth

The stack that fits this corpus is hybrid retrieval into a reranker. Set the Knowledge Base retrieve step to a hybrid search type so both branches run, pull a wide top-N (20 to 50 candidates), then apply a reranker in the retrieve configuration to reorder those candidates and keep the top-k (five) for generation. Retrieve wide, rerank narrow. The width is what gives the reranker something to work with; the narrowing is what keeps the context window honest.

Hybrid alone is often enough. If the failures are purely “the exact token never made the shortlist,” fusion fixes that on its own with no added latency and no extra model call, and that should be the first change shipped. Reach for the reranker when the right document is making the candidate set but landing at rank six or fourteen, below the cutoff. That’s a precision-of-ordering problem, and reordering is exactly what the cross-encoder does better than any first-stage score. Ship hybrid, measure, then add the reranker if the ordering is still wrong.

In Bedrock Knowledge Bases the wiring is configuration, not code. The retrieve request carries a vectorSearchConfiguration with overrideSearchType: HYBRID and a numberOfResults set to the wide N; a reranking configuration names the reranker model and the final number of results to return. OpenSearch users can build the same shape by hand with a search pipeline (normalization-processor for the hybrid fusion) and a call to the Bedrock Rerank API over the fused candidates.

The gotchas are worth pricing in. The reranker adds latency and per-query cost, because it’s another model call scoring N documents; N of 50 costs more and runs slower than N of 20, so size N to the smallest window that reliably contains the right answer. If N is too small the reranker has nothing good to promote, and no amount of reranking rescues a candidate set that never included the target. Hybrid score-weighting needs tuning; the dense and sparse distributions aren’t on the same scale, and a bad normalisation can let one branch drown the other. Rerankers have input token limits, which cap how many candidates and how large each chunk can be, so very large chunks force a smaller N. And the order-of-operations rule: don’t rerank to paper over a recall problem. If the right document isn’t in the top-N at all, the fix is retrieval (better chunking, hybrid, a stronger embedding model), not reordering a set that doesn’t contain the answer.

A worked example: getting ERR-4021 to rank one

The query is “ERR-4021 on firmware 2.3.” Under pure dense retrieval, the top five look like this:

Dense-only top-5 (what reaches the model today)
  1. "Common error codes overview"           sim 0.83
  2. "ERR-4020: sensor timeout"               sim 0.82
  3. "ERR-4102: calibration drift"            sim 0.81
  4. "Firmware 2.x upgrade notes"             sim 0.80
  5. "Troubleshooting the thermostat"         sim 0.79
  ...
  14. "ERR-4021: schedule memory fault (fw 2.3)"  sim 0.71   ← the answer, out of reach

The one document that names ERR-4021 sits at rank 14. Its embedding is close to the query’s, but so are a dozen other error-code notes, and dense similarity can’t separate them. Turn on hybrid, and BM25 weights the rare token ERR-4021 heavily wherever it appears literally. The candidate set (top-N of 30) now contains that release note, pulled up by lexical match, alongside the semantic neighbours:

Hybrid top-N (candidate set, N = 30), fused rank
  1. "ERR-4021: schedule memory fault (fw 2.3)"  fused 0.91   ← now in contention
  2. "Common error codes overview"               fused 0.78
  3. "ERR-4020: sensor timeout"                   fused 0.74
  ...

Hybrid already fixed it here, because the exact token was decisive. Where the target lands mid-pack instead, the reranker settles it: the cross-encoder reads the query and each candidate together and scores real relevance, not vector proximity.

Reranked top-k (k = 5), cross-encoder relevance
  1. "ERR-4021: schedule memory fault (fw 2.3)"  rerank 0.97
  2. "Firmware 2.3 release notes"                 rerank 0.61
  3. "ERR-4020: sensor timeout"                   rerank 0.28
  4. "Common error codes overview"                rerank 0.22
  5. "Troubleshooting the thermostat"             rerank 0.19

The right note is now rank one with a wide margin, and the model answers from the document that actually documents the fault. The cost is one hybrid query (no extra latency over dense) plus one rerank call over 30 candidates (tens of milliseconds and a small per-query charge). For a query class that was silently wrong before, that’s a cheap trade.

What’s worth remembering

1. Dense retrieval reads meaning and fumbles exact tokens; the embedding of a code sits in a tight cluster with every other code.
2. BM25 is the opposite instrument: it nails rare tokens and misses paraphrase. Hybrid runs both and fuses the scores.
3. Hybrid is the direct fix for mixed query styles, and it adds essentially no latency over dense alone; ship it first.
4. A cross-encoder reranker reads query and document together, so it scores true relevance far better than comparing two independent vectors.
5. Reranking is the biggest precision lever and the most expensive, because it’s another model call over N candidates; run it only on a shortlist.
6. Retrieve wide, rerank narrow: pull a generous top-N (20 to 50), reorder, keep a small top-k (about 5) for the context window.
7. If the right document isn’t in the top-N at all, that’s a recall problem; fix retrieval, don’t rerank a set that lacks the answer.
8. In Bedrock Knowledge Bases both levers are configuration: search type HYBRID on the retrieve step, a reranker in the retrieve configuration, no bespoke code.

The assistant keeps its strength on paraphrase and stops losing exact-term queries. Hybrid gets the rare token into contention; the reranker puts it on top; the model answers from the document that names the fault instead of the three that don’t.