The situation
A support-knowledge assistant on Amazon Bedrock is answering staff questions from a Knowledge Base built over product manuals, an internal wiki, and a table of parts. It works well enough in demos and then falls over in specific, repeatable ways. A question about error code E-4471 returns a passage about a different code entirely. A question that quotes a manual almost verbatim retrieves a vaguely related section instead of the exact one. An engineer asking about the HX-200 pump gets an answer about the HX-2000, a different product. And once, after the manuals were updated, the assistant kept citing a procedure that had been rewritten a week earlier.
Every one of these is a retrieval failure rather than a generation failure. The model summarises the chunks it was handed, and the chunks were wrong. When the right passage never reaches the prompt, prompt tuning cannot recover it.
So the useful question is which stage of retrieval produced the wrong passage. Retrieval has a handful of distinct failure modes, each with a recognisable signature, and the diagnosis is mostly a matter of reading the signature.
What actually matters
“Wrong chunk” is at least three different bugs, and telling them apart takes different tests. The right passage might not be in the index at all. It might be in the index but score poorly against the query, so it never enters the candidate set. Or it might enter the candidate set and rank below the cut-off, so it is fetched and thrown away. These are Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks., similarity, and ranking failures, and they sit at different stages of the pipeline. Working back from the answer to the passages that produced it separates them in a single query. A Bedrock Knowledge Base returns five results by default and numberOfResults accepts up to 100, so ask for the maximum and look for the passage you expected. If it appears at rank forty, it was always retrievable, and the problem is ranking or the cut-off, so RerankingA second pass that re-scores a wide set of retrieved candidates and keeps only the few most relevant, so the expensive model reads less. or a larger k is the lever. If it never appears at 100, the problem is upstream: the embedding does not place the query near the document, or the passage is not in the index.
The embedding is only as good as what went into it. A chunk that packs five unrelated topics produces one averaged vector that represents none of them sharply, so it matches broadly and precisely nothing. A chunk cut so small that it loses its surrounding context embeds a fragment that no longer means what the whole meant, so a pronoun-heavy sentence with the subject three lines up retrieves for the wrong thing. Chunk size sets what each vector can represent.
Semantic similarity and exact-token matching are different tools, and a lot of “wrong chunk” cases are a semantic system doing a keyword job. Error codes, SKUs, part numbers, version strings, and API method names carry meaning in their exact characters, and an embedding model places E-4471 and E-4470 close together because they read almost the same. Semantic search generalises across surface differences, which is the wrong behaviour when the surface is the signal. Hybrid search covers that: run a keyword match alongside the vector search and fuse the results, so an exact match on the string contributes to the score.
Retrieval scope is a correctness property. If the index mixes tenants, product lines, or document versions and nothing filters them at query time, the nearest vector might come from the wrong tenant or a superseded manual, semantically perfect and still wrong. Metadata filtering enforces that slice before similarity is considered. Freshness is the same kind of problem: an index reflects the documents as of the last sync, so if the source changed and no ingestion job ran, retrieval returns stale content, and no similarity setting fixes that.
What we’ll filter on
- Recall stage. Is the expected passage even in the index, and does it come back when the candidate count is widened far past the normal cut-off?
- Embedding fidelity. Does each chunk represent a single coherent idea, or is the vector averaging several topics or missing the context that gives a fragment its meaning?
- Index agreement. Do the index’s dimension count, data type, and distance space match what the embedding model actually emits?
- Lexical exactness. Does the query hinge on exact tokens (codes, SKUs, versions) that semantic similarity will smear together?
- Scope correctness. Are results confined to the tenant, product, and document version the query is entitled to, or is the corpus unfiltered?
- Freshness. Does the index reflect the current source documents, and were all of its vectors written by the same embedding model version?
The landscape
Index configuration mismatch. The vector index has to match what the embedding model emits, in three respects: dimension count, data type, and distance space. Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) outputs 1,024 dimensions by default, with 512 and 256 also selectable, and its normalize request parameter defaults to true, so the vectors come back at unit length. That last detail settles an argument people spend a lot of time on. For unit-length vectors, Cosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”., inner product and Euclidean distance all produce the same ordering, which is why AWS’s own instructions for hand-building an OpenSearch index for a Knowledge Base say to use l2 for floating-point embeddings. The space only changes the ranking when the vectors are not unit length, because normalize was turned off or a self-hosted embedder does not normalise, and binary embeddings are a separate case needing a binary data type and the hamming space. Two harder configuration errors sit alongside it: a dimension count that disagrees with the model fails ingestion outright, and an OpenSearch Serverless index built on the nmslib engine rather than faiss cannot serve metadata filters at all, which surfaces later as a scope failure. When Bedrock Knowledge Bases creates the index for you, all of this is set; it creeps in when you bring your own index, or swap embedding models without rebuilding the index the old vectors were written for.
Chunks too large. A chunk covering half a manual page embeds as one vector that is the average of everything in it. The single sentence that answers the query is one signal among many, and averaging drowns it, so the chunk matches many queries weakly and the precise one poorly. The symptom is retrieval that returns the right general area but never the sharp answer, and answers that feel padded because the model is summarising a broad chunk. The fix is smaller, more focused chunks; the chunking-strategy choice is its own decision, covered in choosing a chunking strategy for Bedrock Knowledge Bases.
Chunks too small. Cut too fine and a chunk loses the context that gave it meaning. A step that reads “then set it to 40 psi” embeds without the “it” ever being resolved, so it retrieves for pressure questions in general and not for the pump it belongs to. The symptom is fragments that are individually retrievable but useless in isolation, and answers missing the qualifier that lived in the sentence before. The fix is larger or overlapping chunks, or a hierarchical strategy that keeps a parent’s context attached to each child.
Query phrased unlike the documents. Users ask “why won’t it turn on” while the manual says “unit fails to initialise on power-up.” Semantic search closes some of that gap but not all of it, and short or jargon-light queries land far from formally written source. The symptom is that short, colloquial questions miss while verbose, well-phrased ones hit. The fixes live in query preprocessing, the family of transforms applied between the user pressing enter and the search running: normalising and expanding the query, rewriting it into the register of the source text, decomposing a compound question into sub-queries, and lifting known filters out of the text into metadata. Hybrid search covers what rewriting cannot, and it is also where custom scoring lives, because the fusion step has to normalise the BM25 and vector score distributions before it can weight their contributions; a badly set weighting lets one branch drown the other, so a lexical index added to improve retrieval ends up burying the semantic hits.
Exact-match tokens. This is the E-4471 and HX-200 case. Codes and identifiers carry meaning in their exact characters, and embeddings place near-identical strings at near-identical points, so the wrong code or the wrong SKU ranks first. The symptom is precise: queries built around an identifier fail while prose queries succeed. The fix is hybrid search, which runs a keyword match beside the vector search and fuses the scores; the mechanics are in hybrid search and reranking for Bedrock RAG.
Missing metadata filtering. The index holds several tenants or product lines and the query does not constrain to one, so the nearest neighbour is from the wrong slice. The symptom is answers that are on-topic but from the wrong product, tenant, or version, like the HX-2000 answer to an HX-200 question when both manuals are in one index. The fix is attaching metadata at ingestion (a .metadata.json file sharing the source file’s name and extension) and applying a metadata filter at retrieval so only the entitled slice is searched.
No reranking. The right chunk is retrieved into the candidate set but sits at rank twelve while k is five, so it is fetched and discarded before the model sees it. This is the failure the widen-k test exposes instantly. The symptom is that the answer exists in the corpus and shows up when you ask for more results, but not in the normal cut. The fix is a reranking step: retrieve a wider candidate set, then reorder it with a Cross-encoderA model that reads a query and a passage together and scores the pair, more accurate than comparing two independently-made vectors. reranker, and keep the top few. Bedrock offers two, Amazon Rerank 1.0 (amazon.rerank-v1:0) and Cohere Rerank 3.5 (cohere.rerank-v3-5:0), each in a handful of Regions. Reranking scores query and passage together rather than comparing two independent embeddings, so it recovers the right chunk from deep in the candidate list.
Stale index. The source changed and the ingestion job did not re-run, so retrieval returns the old content. The symptom is answers that were correct and are now citing superseded procedures, with no pattern by query type; it is purely temporal. The fix is re-syncing the data source (syncing is incremental, covering documents added, modified or deleted since the last sync) and, if this recurs, automating the sync so the index does not drift behind the source.
Embedding drift. The index is current, every document synced, and ranking is still nonsense for one slice of the corpus. Somewhere along the way the embedding model version changed underneath the pipeline, so part of the corpus was written by one model and the rest by another, and the search is now comparing vectors from two different spaces. The symptom is a clean split by ingestion date. Documents indexed before the change rank sensibly against each other, documents indexed after do the same, and a query lands in whichever space its own embedder belongs to while the other half is effectively invisible. You can confirm it by re-embedding a sample of known-good passages and checking that a query still ranks them where it used to. Tracking the score distribution of a fixed probe set over time catches it earlier, with an alert when the mean similarity moves. The only real fix is re-embedding the whole corpus with a single pinned model version, and the prevention is recording that version as a property of the index, beside the dimension count and the distance space. Which model to pin to is its own decision, covered in picking an embedding model for retrieval.
Evaluation
Side by side
Reading the symptom is most of the diagnosis. The table maps each cause to whether the right chunk is in the store, whether it comes back when you widen k far past the cut-off, whether the failure is specific to exact-token queries, and whether the fix requires re-ingesting or re-embedding the corpus rather than a query-time change.
| Cause | Right chunk in the store | Comes back at high k | Exact-token queries only | Fix needs re-ingest |
|---|---|---|---|---|
| Index configuration mismatch | ✓ | ✗ (ranking skewed everywhere) | ✗ | ✓ (rebuild index) |
| Chunks too large | ✓ (diluted) | ✗ | ✗ | ✓ |
| Chunks too small | ✓ (fragment) | ✗ | ✗ | ✓ |
| Query unlike documents | ✓ | sometimes | ✗ | ✗ |
| Exact-match tokens | ✓ | ✗ | ✓ | ✗ |
| Missing metadata filter | ✓ (wrong slice ranks) | ✗ | ✗ | ✗ (add metadata, then filter) |
| No reranking | ✓ | ✓ | ✗ | ✗ |
| Stale index | ✗ (old content only) | ✗ | ✗ | ✓ (re-sync) |
| Embedding drift | ✓ (wrong vector space) | ✗ | ✗ | ✓ (re-embed) |
The single most useful column is “comes back at high k.” A yes points almost uniquely at ranking, so reranking or a larger k is the fix. A no sends you upstream to embedding, metric, scope, or freshness, and the remaining columns split those apart.
The same logic drawn as a decision tree: start at the symptom, answer each test, and arrive at the fix.
The solution
Start every investigation with the widen-k test. It needs one query and no configuration change, and it halves the search space. Pull a failing query, request the full hundred results, and scan for the passage you expected. If it is sitting at rank thirty, the corpus and the embeddings are fine, and this is a ranking problem: add a reranker over a wider retrieval, or raise k if the generation budget allows. If it is nowhere at a hundred, stop tuning k. The query and the document do not embed near each other, or the passage is stale or filtered out, and no ranking change recovers it. That branch ends in rechunking, query preprocessing, or a re-embed, all of which mean rewriting the index.
For a configuration mismatch, the tell is that everything is slightly off rather than one query class failing. Check the three things the index and the model have to agree on: dimensions, data type, and space. With Titan Text Embeddings V2 at its defaults, the vectors are unit length and l2 is the space AWS recommends, so a cosine-versus-Euclidean argument is usually a distraction; the case that does change ranking is unnormalised vectors, or binary embeddings indexed as floats. Quick-creating the vector store through Bedrock sets all of it, so suspect this when someone hand-built the index, or when the embedding model was changed without re-embedding, because old vectors compared under a new model’s assumptions rank nonsensically. The fix is rebuilding the index and, if the model changed, re-embedding the corpus, since vectors from two models are not comparable.
For exact tokens, the fix is hybrid search, and it is worth being precise about why pure semantic search cannot be tuned into doing this. An embedding compresses a string into a dense vector that captures meaning and discards surface form, so E-4471 and E-4470 collapse toward the same point. Hybrid search keeps a lexical index alongside the vector index and fuses the two rankings, so an exact keyword hit on the code lifts the correct passage regardless of the embedding score. In Bedrock Knowledge Bases this is the HYBRID value of overrideSearchType rather than SEMANTIC, selectable per retrieval. It has a prerequisite worth checking before you plan around it: hybrid is supported only on OpenSearch Serverless, Aurora (RDS) and MongoDB Atlas vector stores that contain a filterable text field. On any other store the request falls back to semantic search.
For scope errors, metadata filtering is both the fix and a design lesson: relevance is scoped, and the scope must be data the retriever can filter on. Attach the tenant, product line, and version as metadata at ingestion, in a .metadata.json file named after the document it belongs to, then filter on those fields at query time. Without the metadata in place first there is nothing to filter, so this is the one fix that can require re-ingesting to add the fields even though the filtering itself happens at query time.
Reranking separates retrieval recall from final precision. Let the vector search return a wide candidate set tuned for recall, then let a cross-encoder read query and passage together and reorder, so final precision comes from a model that compares the pair rather than from the distance between two vectors computed in isolation. The Rerank API takes one query against up to 1,000 source documents per request, which is far more headroom than a hundred retrieved chunks needs. This is the biggest single improvement available when the widen-k test keeps turning the chunk up deep in the list.
Symptom-reading misses one dimension, which is speed: a retriever can hand back exactly the right chunk and still be why the assistant feels sluggish. In an HNSW index the build-time parameters are m, the number of neighbours each node keeps, and ef_construction, with ef_search setting how wide the graph walk stays at query time. Raising any of them improves recall and costs query latency and memory. How much of that you can reach depends on the store: a hand-built OpenSearch index exposes all three, while OpenSearch Serverless scales capacity itself and exposes far less. Measure recall against a fixed query set at each setting rather than guessing, because the trade-off only resolves once you know how much recall these answers need.
Worked example
An engineer asks, “what does error E-4471 mean on the HX-200,” and the assistant explains E-4470, a different fault on a different pump. Two symptoms in one query, so run the tree.
Widen k to a hundred and search the candidates for the E-4471 passage. It appears, at rank sixty-three, alongside a cluster of E-447x codes that all embed close together, and the top results are E-4470 and E-4472 from the HX-2000 manual. That tells us three things at once. The right chunk is in the store, so this is not a recall, chunking, or freshness problem. It comes back only at high k because the near-identical codes crowd it out, which is the exact-token signature. And the HX-2000 passages ranking at all means the HX-200 query is not scoped to its product, which is the missing-filter signature.
The fix is two query-time changes, no re-embedding. Set overrideSearchType to HYBRID so a keyword match on the literal string E-4471 fuses with the vector score and pulls the exact code up from rank sixty-three. Add a metadata filter on product = HX-200 so the HX-2000 manual is excluded before ranking begins, which removes the wrong-product answers and clears space for the right one. With the corpus and embeddings untouched, the correct E-4471 passage for the HX-200 now lands at the top.
Had the widen-k test turned up nothing at a hundred, the story would be different: the passage would be missing (a stale index needing re-sync) or diluted into an oversized chunk needing a finer chunking strategy, and no amount of hybrid search or filtering would have helped, because you cannot rerank a chunk that never entered the candidate set.
What’s worth remembering
- “Wrong chunk” is at least three separate bugs: the passage is not in the index (recall), it scores poorly against the query (similarity), or it is retrieved and then ranked below the cut-off (ranking). Each needs a different fix.
- The widen-k test splits that space in one query: ask for the maximum hundred results and look for the expected passage. If it appears deep in the list, the problem is ranking; if it never appears, the problem is upstream.
- Exact tokens such as error codes, SKUs, and version strings are a keyword job, and embeddings place near-identical strings together. Hybrid search fuses a lexical match with the vector score, where the vector store supports it.
- Retrieval scope is a correctness property. Attach tenant, product, and version as metadata at ingestion and filter at query time, or a semantically perfect neighbour from the wrong slice ranks first.
- Query preprocessing, hybrid search, metadata filters and reranking are query-time changes you can reverse; rechunking, re-embedding and rebuilding the index all mean rewriting the corpus. Read the symptom before you rebuild anything.
- An index can be perfectly current and still wrong. If the embedding model version changed underneath it, vectors from two models are being compared, only a full re-embed fixes it, and recording the version as a property of the index stops it recurring.