Hybrid Search and Reranking for Bedrock RAG

The situation

The internal support assistant answers questions over product manuals, firmware release notes, and a decade of resolved tickets. It runs a Bedrock Knowledge Base with pure semantic retrieval: embed the query, pull the top five chunks by vectorVectorAn ordered list of numbers – in AI usage, almost always an embedding – and by extension the databases that index them for nearest-neighbour search. similarity, hand them to the modelModelA trained set of weights plus the architecture that makes them useful – the thing you load up and run inference against.. On conceptual questions (“how do I reset the thermostat schedule?”) it works well. Paraphrase is its strength, and the embedding model handles it well.

The complaints are all the same shape. A field engineer types “ERR-4021 on firmware 2.3” and gets back three chunks about other error codes, a general troubleshooting overview, and one paragraph that mentions firmware 2.x in passing. The one release note that documents ERR-4021 specifically is sitting at rank 14, outside the window that ever reaches the model. The answer the assistant generates reads as confident, is groundedGroundingConstraining a model to answer from provided sources rather than from whatever it absorbed during training. in the wrong chunks, and is wrong.

The pattern is exact-term queries. Product codes, error codes, part numbers, acronyms, proper names. The tokens that carry the whole meaning of the query are precisely the tokens dense retrieval smears together. Retrieval precision on that slice of traffic needs to come up.

What actually matters

A dense retriever embeds the query and each chunk into the same space with a bi-encoder, then compares the two vectors. That comparison is why paraphrase works: “reset the schedule” and “clear the programmed times” land near each other even with no shared words. It is also why exact terms fail. ERR-4021 has almost no semantic content of its own; its embedding is dominated by the pattern “an error code,” so it sits in a tight cluster with ERR-4020, ERR-4102, and every other code the corpus has ever seen. The vectors that should be far apart are close, and a similarity score does not separate them.

Sparse retrieval is the opposite instrument. BM25 scores documents by exact token overlap, weighted by how rare each token is across the corpus. A rare token like ERR-4021 gets a high weight the moment it appears, so the one document containing it shoots to the top. The trade-off is that BM25 scores no overlap at all between “reset the schedule” and “clear the programmed times”, because they share no tokens. It matches strings, not meaning.

Hybrid search runs both and fuses the scores. The exact-term query gets BM25’s precision on the rare token; the paraphrase query gets the embedding model’s semantic reach; a mixed query gets a blend. Fusion is where the tuning lives, normalising two score distributions that aren’t on the same scale and weighting their contributions.

Fusion lifts the right document into contention without guaranteeing rank one, and reordering is a second stage. A first-stage retriever, dense or sparse, scores every candidate independently: it embeds the query once, embeds each document once, and compares. A cross-encoder reranker instead reads the query and one candidate document together in a single pass and scores their relevance directly. Reading both together, it picks up signals that two separately computed vectors never encode. It is more precise than any first-stage score and it needs a model call per batch of candidates, so it only ever runs over a shortlist.

That fixes the lever order. Chunking decides what a document even is; retrieval method (dense, sparse, hybrid) decides what makes the shortlist; the reranker reorders the shortlist by true relevance; the top of the reordered list goes to the model. Retrieve wide, rerank narrow: pull a generous top-N so the right document is somewhere in the candidates, then let the reranker promote it into the small top-k that fits the context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined..

What we’ll filter on

1. Exact-term queries, does the method surface rare tokens (codes, part numbers, proper names)?
2. Paraphrase, does it still handle semantically-similar-but-differently-worded queries?
3. Final precision, how good is the small top-k that actually reaches the model?
4. Added latency per query, what does the method add to p99 retrieval time?
5. Added cost per query, extra model or index calls per request?
6. Managed availability, is it a first-class Bedrock or OpenSearch feature or bespoke plumbing?

The landscape

1. Dense / vector-only search. The baseline the assistant already runs. A bi-encoder embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model maps query and chunks into one space; retrieval is approximate-nearest-neighbourANNIndex structures (HNSW graphs, IVF partitions) that answer the k-nearest-neighbours question fast by giving up guaranteed exactness – recall becomes a tunable knob rather than a certainty. over the chunk vectors. Strong recall on paraphrase and conceptual questions, weak on exact tokens. No extra latency beyond the one ANN lookup, no extra cost beyond the query embedding. It is the thing to improve, not the answer.

2. Sparse / keyword (BM25). Classic lexical retrieval, scoring by rare-token overlap. Nails exact terms, misses paraphrase entirely. Available as a plain OpenSearch match query. On its own it trades one failure mode for the opposite one, so it’s a component, not a destination.

3. Hybrid search. Run dense and sparse together and fuse. Amazon OpenSearch supports this directly: a hybrid query with a search pipeline whose normalization processor rescales the BM25 and k-NN score distributions (min_max or l2) and combines them by weighted arithmetic, geometric, or harmonic mean, in one round trip. Amazon Bedrock Knowledge Bases exposes the same idea as an overrideSearchType on the retrieve step, set to SEMANTIC or HYBRID. Hybrid there is available only on Amazon RDS, Amazon OpenSearch Serverless, and MongoDB vector stores that contain a filterable text field; every other store runs semantic search whatever you set, and with the field unset the search type follows the vector store configuration. Hybrid is the direct answer to a corpus with mixed query styles, and both branches run inside the one retrieve request.

4. Reranking. A second stage, not a retriever. The Amazon Bedrock Rerank API takes exactly one query and up to 1,000 source documents, and returns them reordered with a relevance score on each. Two reranker models are offered: Amazon Rerank 1.0 (amazon.rerank-v1:0) and Cohere Rerank 3.5 (cohere.rerank-v3-5:0). Neither is in every Region, and Amazon Rerank 1.0 is absent from US East (N. Virginia), where Cohere Rerank 3.5 is the only choice, so confirm availability before designing around one. Bedrock Knowledge Bases applies the same models inside the retrieve step through a rerankingConfiguration, reordering chunks before generation. Reranking is the biggest precision lever here and the one that carries a per-query charge, because it is another model call over the candidate set. It works on text only.

5. Query expansion. A Bedrock call that broadens the query before it is embedded and before the keyword half of a hybrid search runs: synonyms, expanded acronyms, product codenames, adjacent phrasing. “Thermostat forgets its times” goes in and comes back carrying “schedule memory”, “programmed schedule cleared”, “settings lost after power cycle”, so BM25 has something rare to match and the dense branch embeds a richer sentence. It raises recall on under-specified queries, the ones where neither hybrid search nor a reranker helps because the discriminating terms are nowhere in what the user typed. It adds one model call per query, and it can drift the query away from what was asked, which is why the expanded terms belong in the sparse half of a hybrid query rather than replacing the original.

6. Query decomposition. Bedrock Knowledge Bases can split a multi-part question (“compare ERR-4021 and ERR-4102 behaviour on firmware 2.3”) into sub-queries, retrieve for each, and merge. It is a RetrieveAndGenerate option rather than something you assemble yourself, set through orchestrationConfiguration.queryTransformationConfiguration with the type QUERY_DECOMPOSITION. It raises recall on compound questions rather than precision on a single term, so it’s complementary, not a substitute.

7. Metadata filtering. An orthogonal precision lever: restrict candidates by structured attributes (product line, firmware version, document type) before or after the vector match. It narrows the candidate set with no extra model call when the query carries a hard constraint, and it stacks with any of the above. Knowledge Bases offers equals, notEquals, the four numeric comparisons, in, notIn, and the string and list contains operators, combined with andAll or orAll over groups of up to five filters. Some operators are store-dependent, startsWith being OpenSearch Serverless only.

Where the extra query work runs matters as much as whether it happens. One call, a single expansion or a single decomposition, fits in a Lambda function in front of retrieval: one invocation, one model call, one merged result set. Once the handling becomes multi-step and branching (expand, then decompose the expanded query, then pull a metadata filter out of the parts), a Step Functions state machine is the better fit, because every step is separately retryable and every transition is traceable. Sophisticated query handling systems are built out of three techniques that get conflated constantly, so it’s worth keeping them apart: query expansion adds terms to a single question; query decomposition splits one question into several and retrieves for each; query transformation rewrites the question into a different shape, such as a metadata filter plus a semantic search over what is left. Each adds a call ahead of retrieval, so add one where it lifts retrieval effectiveness on a slice of traffic you have actually measured.

Evaluation

Side by side

No single row is the whole answer. Hybrid fixes what dense misses on exact terms; the reranker fixes what any first-stage ranking leaves in the wrong order. For this corpus the two stack: hybrid gets ERR-4021 into the candidate set, the reranker gets it to rank one.

The retrieval pipeline

The solution

The stack that fits this corpus is hybrid retrieval into a reranker. Set the Knowledge Base retrieve step to a hybrid search type so both branches run, pull a wide top-N (20 to 50 candidates), then apply a reranker in the retrieve configuration to reorder those candidates and keep the top-k (five) for generation. Retrieve wide, rerank narrow. The width is what gives the reranker something to work with; the narrowing is what keeps the context window small.

Hybrid alone is often enough. If the failures are purely “the exact token never made the shortlist,” fusion fixes that on its own with no extra model call, and that should be the first change shipped. Reach for the reranker when the right document is making the candidate set but landing at rank six or fourteen, below the cutoff. That’s a precision-of-ordering problem, and reordering is what the cross-encoder does better than any first-stage score. Ship hybrid, measure, then add the reranker if the ordering is still wrong.

In Bedrock Knowledge Bases the wiring is configuration, not code. The retrieve request carries a vectorSearchConfiguration with overrideSearchType: HYBRID and a numberOfResults set to the wide N, which accepts 1 to 100 and defaults to 5. A rerankingConfiguration alongside it names the reranker model ARN and a numberOfRerankedResults, also 1 to 100, for the final count. OpenSearch users can build the same shape by hand with a search pipeline for the fusion and a call to the Bedrock Rerank API over the fused candidates.

The reranker adds latency and a charge, because it is another model call over the candidate set. Reranking is billed per query, and a query is one call carrying up to 100 document chunks, so N of 20 and N of 50 fall inside the same billing unit while N of 150 does not. Latency still grows with N, so size N to the smallest window that reliably contains the right answer. If N is too small the reranker has nothing better to promote, and no amount of reranking rescues a candidate set that never included the target. Hybrid score-weighting needs tuning; the dense and sparse distributions aren’t on the same scale, and a bad normalisation lets one branch dominate the fused score. The candidate set has hard ceilings either way, at 100 chunks per knowledge base retrieve and 1,000 sources per direct Rerank call. And the order-of-operations rule: don’t rerank to paper over a recall problem. If the right document isn’t in the top-N at all, the fix is retrieval (better chunking, hybrid, a stronger embedding model), not reordering a set that doesn’t contain the answer.

Worked example

The query is “ERR-4021 on firmware 2.3.” Under pure dense retrieval, the top five look like this:

Dense-only top-5 (what reaches the model today)
  1. "Common error codes overview"           sim 0.83
  2. "ERR-4020: sensor timeout"               sim 0.82
  3. "ERR-4102: calibration drift"            sim 0.81
  4. "Firmware 2.x upgrade notes"             sim 0.80
  5. "Troubleshooting the thermostat"         sim 0.79
  ...
  14. "ERR-4021: schedule memory fault (fw 2.3)"  sim 0.71   ← the answer, out of reach

The one document that names ERR-4021 sits at rank 14. Its embedding is close to the query’s, but so are a dozen other error-code notes, and the dense similarity scores don’t separate them. Turn on hybrid, and BM25 weights the rare token ERR-4021 heavily wherever it appears literally. The candidate set (top-N of 30) now contains that release note, pulled up by lexical match, alongside the semantic neighbours:

Hybrid top-N (candidate set, N = 30), fused rank
  1. "ERR-4021: schedule memory fault (fw 2.3)"  fused 0.91   ← now in contention
  2. "Common error codes overview"               fused 0.78
  3. "ERR-4020: sensor timeout"                   fused 0.74
  ...

Hybrid already fixed it here, because the exact token was decisive. Where the target lands mid-pack instead, the reranker changes the order: the cross-encoder reads the query and each candidate together and scores relevance directly, not vector proximity.

Reranked top-k (k = 5), cross-encoder relevance
  1. "ERR-4021: schedule memory fault (fw 2.3)"  rerank 0.97
  2. "Firmware 2.3 release notes"                 rerank 0.61
  3. "ERR-4020: sensor timeout"                   rerank 0.28
  4. "Common error codes overview"                rerank 0.22
  5. "Troubleshooting the thermostat"             rerank 0.19

The right note is now rank one with a wide margin, and the model answers from the document that actually documents the fault. That takes one hybrid query, with both branches inside a single retrieve request, plus one rerank call over 30 candidates, which stays inside a single billed rerank query because it is under 100 chunks. For a query class that was wrong with no error to show for it, one extra model call is a trade worth making.

What’s worth remembering

1. Dense retrieval matches meaning and misses exact tokens; the embedding of a code sits in a tight cluster with every other code.
2. BM25 is the opposite instrument: it nails rare tokens and misses paraphrase. Hybrid runs both and fuses the scores.
3. Hybrid is the direct fix for mixed query styles, and Bedrock offers it only on RDS, OpenSearch Serverless and MongoDB stores with a filterable text field; ship it first where you can.
4. Reranking is the biggest precision lever and the one with a per-query charge, billed per call of up to 100 chunks; run it only on a shortlist.
5. Retrieve wide, rerank narrow: pull a generous top-N (20 to 50), reorder, keep a small top-k (about 5) for the context window.
6. If the right document isn’t in the top-N at all, that’s a recall problem; fix retrieval, don’t rerank a set that lacks the answer.

The assistant keeps its strength on paraphrase and stops losing exact-term queries. Hybrid gets the rare token into contention; the reranker puts it on top; the model answers from the document that names the fault instead of the three that don’t.