The situation
A support assistant has been in production for eighteen months. It runs an Amazon Bedrock model over a Bedrock Knowledge Base, and the knowledge base sits on an Amazon OpenSearch Serverless vector collection. At launch the index held about 2 million ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. at 1,024 Embedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching., and retrieval came back at a p99 of roughly 90 milliseconds.
Three months ago retrieval p99 was 180 milliseconds. This week it is 540. Nothing broke in between: no deployment changed the retrieval path, no alarm fired, no ingestion job failed. The corpus is now a little over 12 million chunks. The content team connected two more document sources in the spring, and nobody has deleted anything from the index since launch, even though about 900,000 source documents have been archived or superseded on the origin side.
There is a second, quieter complaint. Support leads say the assistant has got vaguer. It used to cite the right policy page; now it sometimes cites a neighbouring one, or an old revision of the same page. Nobody can produce a failing example on demand, which is how quality complaints usually arrive.
A second team inside the same organisation runs its own assistant on Amazon Aurora PostgreSQL with the pgvector extension. Their p99 is fine but their nightly ingestion has started taking four hours instead of forty minutes, and their database CPU sits at 80% for most of it. Both teams have been asked the same thing: what do we watch, and what do we do about it?
What actually matters
Slow and stale retrieval is a symptom with three quite different causes underneath it, and the store reports none of them as a failure. Every query still returns k results, with scores, inside the timeout. An ANNIndex structures (HNSW graphs, IVF partitions) that answer the k-nearest-neighbours question fast by giving up guaranteed exactness – recall becomes a tunable knob rather than a certainty. index degrades by returning slightly worse neighbours, and there is no exception for “these are the wrong passages”. That absence is the operational problem: without a signal you build yourself, the only detector is a support lead’s hunch three months after the drift started.
The first cause is capacity saturation. The store is doing the right work and does not have enough machine to do it in. On OpenSearch Serverless that shows up as search and indexing OCU utilisation climbing toward the collection’s configured ceiling, with request latency following it. On a provisioned OpenSearch domain the list is longer: CPU utilisation, JVM memory pressure past the point where garbage collection eats the query budget, rising SearchLatency and IndexingLatency, FreeStorageSpace trending toward the low-watermark, search thread-pool rejections, and a cluster health colour that has gone yellow. On Aurora with pgvector it looks different again: connection saturation, freeable memory falling until the index no longer sits in the buffer cache, and read IOPS climbing as the query starts touching disk. Saturation is the cheapest cause to diagnose because every one of these signals is published for you.
The second cause is index degradation, and it is the one nobody watches for. An HNSWA graph-based vector index that walks neighbour links to find close vectors fast, at the cost of extra memory per vector. graph built with an m chosen for 2 million vectors is still a valid graph at 12 million; it is just a worse one, with longer traversals and less certain neighbourhoods, and m cannot be changed without a rebuild. Deleted documents leave tombstones in OpenSearch segments that keep occupying graph memory and get traversed on every query until a merge expunges them; 900,000 of them is 7% of the index doing nothing but slowing things down. An IVFA vector index that clusters vectors up front and searches only the nearest clusters – cheaper memory than a graph index, more tuning. centroid set trained on a sample of the corpus as it stood a year ago now partitions a corpus that has drifted away from it. The lists go lopsided, the probed lists hold fewer of the true neighbours, and recall sags without one thing in the logs changing. Degradation produces exactly the pair of symptoms in front of us: slower queries and worse answers, from an index that has not been touched.
The third cause is data quality, and it is the only one that can hurt answers while leaving latency untouched. Vectors whose source document was deleted months ago are still retrievable and still confidently cited. A change of embedding model leaves two incompatible populations in one index if the backfill was partial, and a change of dimension leaves writes silently rejected or an index that cannot be queried consistently. Duplicate chunks from a source connected twice crowd the top-k so one document supplies every result. Chunks whose text extraction failed embed as near-empty vectors that sit suspiciously close to everything. None of this trips a metric. It is found by looking, which is why the looking has to be scheduled.
Getting the diagnosis wrong is expensive in both directions. Adding OCUs or a bigger instance to an index-degradation problem buys a few weeks of headroom at permanent extra cost and leaves recall exactly where it was. Booking a maintenance window to rebuild an index that is merely saturated spends the window and returns the same latency. So the work is to make the three causes separable in evidence before choosing a lever.
What we’ll filter on
- Which cause the evidence points at: saturation, index degradation, or data quality. Each has a distinct fingerprint and they can be present together.
- Whether the signal already exists: does the store publish this metric, or does it have to be built and emitted as a custom one?
- Whether the remedy runs online: can it happen against a live index, or does it need a rebuild, an alias swap, and a window?
- Whether a re-embed is required: a rebuild reuses the vectors you have; a re-embed regenerates them and costs corpus-sized inference.
- Whether it can be automated on a schedule, or whether it needs a person deciding each time.
The landscape
Performance monitoring for vector databases
Start with what each store hands you, because performance monitoring for vector databases is mostly a matter of knowing which published metric belongs to which cause.
OpenSearch Serverless. The collection publishes search and indexing OCU consumption to CloudWatch, along with request counts, request latency, and errors. The signal that matters most is utilisation against the collection’s configured maximum capacity, because a collection pinned at its ceiling queues rather than scales. Serverless hides the cluster internals, so you get consumption and latency but not JVM or segment detail.
Provisioned OpenSearch domains. A domain publishes far more: CPUUtilization, JVMMemoryPressure, SearchLatency, IndexingLatency, FreeStorageSpace, ClusterStatus as green, yellow, or red, ThreadpoolSearchRejected, and the k-NN plug-in’s own statistics including graph memory usage, graph query and index requests, and cache eviction counts. Graph memory usage against the k-NN memory circuit breaker is the one that predicts an index outgrowing its node, because an HNSW graph evicted from memory is served from disk and the latency curve goes vertical.
Aurora PostgreSQL with pgvector. The instance publishes CPUUtilization, DatabaseConnections, FreeableMemory, and read and write IOPS, and Performance Insights attributes load to individual statements so you can see the vector query separately from everything else the database is doing. The vector-specific health lives in the catalogue rather than CloudWatch: dead-tuple counts in pg_stat_user_tables, when autovacuum last ran on the embeddings table, and index size relative to table size for bloat. An ingestion job that slows from forty minutes to four hours with CPU at 80% is usually autovacuum falling behind the write rate, not a query problem at all.
The application side. Whatever the store reports, the number a user feels is retrieval latency measured in your own trace, separate from generation latency. Splitting the two in the application’s own observability is what lets you say retrieval tripled while generation stayed flat.
Automated index optimization routines
Index optimization is a maintenance job, and it wants to run on a cadence rather than after a complaint. The automated index optimization routines available differ by store.
On OpenSearch, the first routine is a force merge that expunges deleted documents, which reclaims the graph memory the tombstones were holding and shortens traversals. The second is a reindex into a freshly built index with parameters sized for the corpus as it is now, published behind an alias so the swap is atomic and reversible. Index State Management policies can schedule the merge. A rebuild is a pipeline you run rather than an operation the store offers, which makes how the index is split an operational decision as much as a performance one: a rebuild’s blast radius is one index.
On pgvector, REINDEX INDEX CONCURRENTLY rebuilds an HNSW or IVFFlat index without taking a write lock on the table, at the cost of running longer and needing space for both copies. Autovacuum settings on a high-churn embeddings table usually need tightening from the defaults, because vacuum is what reclaims dead tuples and keeps the index from bloating past its useful size. VACUUM ANALYZE after a bulk load keeps the planner honest about whether to use the vector index at all.
IVF has its own routine: the centroids are trained once on a sample, so retraining and rebuilding is what corrects for corpus drift. There is no incremental version. HNSW has no equivalent, because it has no training step, but it also has no way to raise m in place, so its rebuild is unavoidable when the graph is undersized for the corpus.
Vector database query optimization for retrieval augmentation
Some of the latency is in the query rather than the index. Vector database query optimization for retrieval augmentation means the levers you pull per request: ef_search on HNSW, nprobe on IVF, the value of k, and whether a metadata filter runs as a pre-filter during traversal or a post-filter afterwards. These are the knobs that trade recall against latency, and unlike a rebuild they take effect on the next query. A post-filter that discards most of a k-of-100 result set is doing a hundred vectors of work to return five, and moving it to a pre-filter is often worth more than a hardware change.
Data quality validation processes
The data quality validation processes worth scheduling are four checks, all of them cheap:
- Reconciliation. Compare the set of source document identifiers against the set of documents in the index and report both directions. Documents in the source and not the index are ingestion failures that never announced themselves; documents in the index and not the source are orphans still being cited.
- Dimension and model assertion. Every vector in an index must have the same dimension and come from the same embedding model. Record the model identifier and dimension as metadata on write, then count distinct values. More than one means the index is mixed.
- Duplicate detection. Hash chunk text and count collisions. A source connected twice, or an ingestion job re-run without a delete, shows up here before it shows up as a monotonous top-k.
- Empty and degenerate chunks. Flag chunks below a length floor, or whose vector norm is an outlier. These are usually failed text extraction, and they behave as universal near-neighbours, which is one route to the wrong passage being retrieved.
A scheduled recall probe
None of the above measures recall, and recall is the thing that actually degraded. Build a probe: a fixed set of query-to-expected-chunk pairs, forty or so, curated once from real questions with the correct passage identified by a human. Run it on a schedule against production, compute recall at k and mean reciprocal rank, and emit both as CloudWatch custom metrics with an alarm on the drop. That turns “the assistant feels vaguer” into a graph with a date on it, and it is the only signal here that separates an index that is slow from an index that is wrong. Keep the pairs stable and version them alongside the ingestion pipeline, since a probe whose expected chunks have been re-chunked underneath it reports a fall that never happened.
Evaluation
Side by side
| Lever | Fixes latency | Restores recall | Runs online | Schedulable | Signal that calls for it |
|---|---|---|---|---|---|
| Add capacity (OCUs, instance, replicas) | ✓ | ✗ | ✓ | ✗ | OCU or CPU at ceiling, JVM pressure, connections saturated |
Tune ef_search, nprobe, k, filter order |
✓ | ✓ | ✓ | ✗ | Latency high with headroom to spare; post-filter discarding most results |
| Force merge to expunge deletes | ✓ | ✓ | ✓ | ✓ | Deleted-document ratio climbing, graph memory above corpus size |
| Reindex behind an alias | ✓ | ✓ | ✗ | ✓ | Corpus several times the size the graph was built for |
| Retrain IVF centroids and rebuild | ✓ | ✓ | ✗ | ✓ | Probe recall falling on an IVF index with a stale training sample |
| Re-embed the corpus | ✗ | ✓ | ✗ | ✗ | Embedding model or dimension changed; mixed populations in one index |
| Reconcile and repair the data | ✗ | ✓ | ✓ | ✓ | Orphans, duplicates, mixed dimensions, or degenerate chunks found |
Two columns carry most of the decision. Adding capacity is the only lever that fixes latency and does nothing at all for recall, which makes it the wrong answer whenever the probe has moved. Re-embedding is the only lever that fixes recall and does nothing for latency, and it is the most expensive one on the list, so it needs a specific trigger rather than a suspicion. Everything in the middle three rows both speeds queries up and improves the neighbours they return, because a smaller, better-shaped graph is doing less work for a better result.
The order of the gates is doing work. Reading saturation first is tempting because those metrics are free, but a saturated store and a degraded index produce the same latency curve, and only the probe tells you which. Checking the embedding model before reconciliation is deliberate too: a mixed-model index makes every other measurement meaningless, because half the vectors are in a different space and no amount of reindexing will bring them into the same one.
The solution
Treat vector store operational management as three standing pieces: an alarm set, a maintenance cadence, and a probe. None of them is optional and none of them substitutes for the others.
The alarm set
On an OpenSearch Serverless collection, alarm on search OCU and indexing OCU utilisation crossing about 70% of the configured maximum, on request latency p99 against a budget you have written down, and on the 4xx and 5xx error counts. Set the collection’s maximum capacity deliberately rather than leaving it at the default, because that ceiling is both a cost control and the thing your utilisation alarm is measured against.
On a provisioned OpenSearch domain, alarm on ClusterStatus leaving green, JVMMemoryPressure above 75%, CPUUtilization sustained above 80%, FreeStorageSpace below the disk low-watermark, SearchLatency p99 against budget, and search thread-pool rejections above zero. Add the k-NN plug-in’s graph memory usage against the circuit-breaker limit, since an evicted graph is the single sharpest latency cliff in this list.
On Aurora with pgvector, alarm on CPUUtilization, DatabaseConnections against the connection limit, FreeableMemory falling toward the size of the index, and read IOPS as a proxy for the index no longer being cached. Add two custom metrics from the catalogue: dead tuples on the embeddings table, and hours since the last autovacuum on it. The four-hour ingestion in this scenario is that second metric, and the fix is autovacuum tuning on that one table, not a bigger instance.
The maintenance cadence
Weekly, run the data quality validation processes: reconciliation both directions, distinct dimension and model count, duplicate hashes, degenerate chunk count. Publish the counts as metrics so a rise is visible before it is a complaint.
Weekly or nightly, depending on delete volume, force merge to expunge deleted documents on OpenSearch, or confirm autovacuum has kept up on pgvector. This is the routine that would have kept 900,000 tombstones out of the graph.
Quarterly, or whenever the corpus grows past roughly twice the size the index was built for, rebuild. Build the new index with parameters chosen for the corpus as it now stands, load it, run the probe against both, and swap the alias only when the new index measures better. An IVF index retrains its centroids as part of that rebuild; an HNSW index gets a larger m and a build-time ef_construction to match. Doing this on a cadence, behind an alias, with the probe as the acceptance test, is what makes it a routine rather than an incident.
The rule about re-embedding
A reindex reuses the vectors you already have and changes only their arrangement. A re-embed regenerates them. Whenever the embedding model or the dimension changes, a re-embed is required and a reindex is not sufficient, because vectors from two models are not comparable even at identical dimension, and distances computed across them are meaningless. Run it as a build into a new index with a new alias target, and keep the old index queryable until the probe passes on the new one. Tag every vector with its model identifier and dimension at write time, so a mixed state is detectable rather than inferred. This is also the reason the store you chose matters operationally: the cost of a full re-embed and swap is a property of the store as much as of the corpus.
Worked example
Take the two teams in turn.
The OpenSearch assistant
The probe goes in first, forty query-to-chunk pairs curated from real support questions. The first run reports recall at 5 of 0.79. There is no history to compare it to, so the team runs it against a snapshot of the index restored from three months ago and gets 0.93. The drift is now a number.
Recall has fallen, so the flow goes down the left. Model and dimension: every vector carries the same model tag and 1,024 dimensions, so no re-embed. Reconciliation is uglier: 912,000 orphaned documents whose source was archived, no dimension mismatches, 4,100 duplicate chunk hashes from a source connected twice in the spring, and 800 chunks under the length floor. The short ones are a batch of scanned PDFs whose text extraction failed.
That is a data quality answer and an index answer together. The repair deletes the orphans and the duplicates, and re-extracts the scanned batch. The deletes leave a fresh pile of tombstones, so a force merge follows, which drops graph memory by about a fifth on its own. Then the rebuild: a new index with m sized for 11 million live chunks rather than 2 million, built and loaded behind a second alias, probed, and swapped when it measures 0.94 at a p99 of 130 milliseconds. No extra OCUs were bought, and the utilisation alarm that never fired was correct not to.
The pgvector assistant
Their probe is flat and their p99 is fine, so the flow goes right at the first gate. Saturation: CPU at 80% during ingestion, connections comfortable, freeable memory falling through the night and recovering by morning. The catalogue tells the rest of it. Dead tuples on the embeddings table are in the millions and autovacuum last completed on that table two days ago, because the nightly upsert churns more rows than the default thresholds trigger on.
The remedy is autovacuum tuning on that one table, a lower scale factor and more workers, plus a REINDEX INDEX CONCURRENTLY on the vector index to reclaim the bloat that has already accumulated. Ingestion returns to under an hour. Nothing about the index parameters was wrong, and a bigger instance would have hidden the problem for another quarter and then met it again.
What’s worth remembering
- An approximate index degrades silently, so recall has to be measured on a schedule with a fixed probe set and emitted as a custom metric; nothing in the store will tell you the neighbours got worse.
- Slow-and-stale retrieval has three causes with disjoint remedies: capacity saturation, index degradation, and data quality. Diagnose before you choose, because capacity fixes latency only and re-embedding fixes recall only.
- Deleted documents keep occupying graph memory and query time until a merge expunges them, which makes force merge on OpenSearch, and autovacuum on pgvector, the cheapest recurring win available.
- An HNSW
mand an IVF centroid set are chosen for a corpus size and shape; when the corpus outgrows them the only remedy is a rebuild, run behind an alias with the probe as the acceptance test. - Whenever the embedding model or the dimension changes, re-embed rather than reindex, and tag every vector with its model and dimension so a mixed index is detectable.
- Weekly reconciliation in both directions, source against index, catches orphans and ingestion gaps that no latency or utilisation metric will ever surface.