Exam-style
A Bedrock Knowledge Base sits on an OpenSearch Serverless collection. Over four months the corpus has grown fourfold with heavy document churn, thousands of updates a day replacing existing pages. Search latency at p99 has tripled, OCU utilisation peaks around forty per cent, every ingestion job reports success, and recall against a fixed probe set of 200 questions has fallen from 96 to 88 per cent. What should the team do?
Reveal the answer
D. Rebuild or force-merge the index with build parameters chosen for the corpus as it is now, then schedule a recall probe against the fixed question set so the measurement runs weekly rather than arriving as a complaint
Three things go wrong with a vector store in production, and each has a metric that identifies it. Capacity saturation shows up in OCU utilisation, queue depth and throttles; at forty per cent peak, the resource is not the constraint here. Data quality problems show up in ingestion failures, stale-document age and orphaned vectors; every job is green, so the corpus is arriving. That leaves index degradation, which is the one nothing reports on its own: an HNSW graph built for a corpus a quarter of the present size, carrying four months of tombstones from documents that were replaced rather than added. Deleted vectors stay in the graph as skipped nodes until a merge removes them, so every search walks further to find the same neighbours, which is the tripled latency, and the connectivity the graph was built with no longer suits the density it now holds, which is the lost recall. A rebuild or force-merge with revised parameters, m and ef_construction picked for the corpus as it stands, fixes both at once. The scheduled probe matters as much as the rebuild, because recall drifts slowly and users report it as vague dissatisfaction months later. Of the rest: added capacity buys nothing when utilisation already says the resource is spare. A smaller top-k trims latency by retrieving less, which trades away the recall that is already the complaint. A larger embedding model is an expensive answer to a diagnosis nobody has made, since a scoring problem would have shown up as poor recall from day one rather than as a slide from 96 per cent. Raising ef_search alone buys some recall back by making every search do more work, on a system whose latency is the presenting symptom, and leaves the degraded graph in place to keep degrading.
Q. p99 search latency has tripled on a Knowledge Base collection, OCU utilisation peaks at forty per cent, ingestion is green, and recall on a fixed probe set has gone from 96 to 88 per cent. What now?
A. Rebuild or force-merge the index with build parameters sized for the corpus it holds today, and put the recall probe on a schedule so the next slide is a graph rather than a complaint.
Why? A vector store fails in three ways and each has its own tell. Capacity saturation reads on OCU utilisation, queues and throttles. Data quality problems read on ingestion failures, document age and vectors left behind by deletes, which is what data quality validation processes exist to catch before retrieval quality does. Index degradation reads on nothing you get for free, which is why performance monitoring for vector databases has to include a recall probe: a fixed question set with known-good passages, run on a timer, graphed next to latency. Here the first two are clean, so the third is the diagnosis. Four months of churn on a graph built for a quarter of the current corpus leaves tombstoned nodes the traversal walks past on every search, and a neighbour count chosen for a sparser index, so latency climbs and recall falls together. That combination is the signature. Rebuilding is the fix, and automated index optimization routines, merges on a schedule and a rebuild triggered when the probe drops through a threshold, are how it stops being an emergency. The parameters you build the graph with are a decision made once against a corpus size, and a corpus that grows fourfold has outrun them, which is the same reason the store you chose at the start deserves a second look once the shape of the data is known rather than guessed.