Keeping a Vector Store Healthy in Production

The situation

A support assistant has been in production for eighteen months. It runs an Amazon Bedrock model over a Bedrock Knowledge Base, and the knowledge base sits on an Amazon OpenSearch Serverless vector collection. At launch the index held about 2 million chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. at 1,024 dimensionsEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching., and retrieval came back at a p99 of roughly 90 milliseconds.

Three months ago retrieval p99 was 180 milliseconds. This week it is 540. Nothing broke in between: no deployment changed the retrieval path, no alarm fired, no ingestion job failed. The corpus is now a little over 12 million chunks. The content team connected two more document sources in the spring, and nobody has deleted anything from the index since launch, even though about 900,000 source documents have been archived or superseded on the origin side.

There is a second, quieter complaint. Support leads say the assistant has got vaguer. It used to cite the right policy page; now it sometimes cites a neighbouring one, or an old revision of the same page. Nobody can produce a failing example on demand, which is how quality complaints usually arrive.

A second team inside the same organisation runs its own assistant on Amazon Aurora PostgreSQL with the pgvector extension. Their p99 is fine but their nightly ingestion has started taking four hours instead of forty minutes, and their database CPU sits at 80% for most of it. Both teams have been asked the same thing: what do we watch, and what do we do about it?

What actually matters

Slow and stale retrieval is a symptom with three quite different causes underneath it, and the store reports none of them as a failure. Every query still returns k results, with scores, inside the timeout. An approximate-nearest-neighbourANNIndex structures (HNSW graphs, IVF partitions) that answer the k-nearest-neighbours question fast by giving up guaranteed exactness – recall becomes a tunable knob rather than a certainty. index degrades by returning slightly worse neighbours, and there is no exception for “these are the wrong passages”. That absence is the operational problem: without a signal you build yourself, the only detector is a support lead’s hunch three months after the drift started.

The first cause is capacity saturation. The store is doing the right work and does not have enough machine to do it in. On OpenSearch Serverless that shows up as SearchOCU and IndexingOCU climbing toward the maximum capacity configured for the account or the collection group, with request latency following. On a provisioned domain the list is longer, running from CPU and JVM pressure through storage headroom to search thread-pool rejections and a yellow cluster. On Aurora with pgvector it is connection saturation, and memory falling until the index is no longer served from cache. Saturation is the easiest cause to diagnose, because every one of these signals is published for you.

The second cause is index degradation, and it is the one nobody watches for. An HNSWHNSWA graph-based vector index that walks neighbour links to find close vectors fast, at the cost of extra memory per vector. graph built with an m chosen for 2 million vectors is still a valid graph at 12 million; it is just a worse one, with longer traversals and less certain neighbourhoods, and m cannot be changed without a rebuild. Deleted documents leave tombstones in OpenSearch segments that keep occupying graph memory and get traversed on every query until a merge reclaims them; 900,000 of them is 7% of the index doing nothing but slowing things down. An IVFIVFA vector index that clusters vectors up front and searches only the nearest clusters – cheaper memory than a graph index, more tuning. centroid set trained on last year’s sample now partitions a corpus that has drifted away from it, so the probed lists hold fewer of the true neighbours and recall sags with nothing changing in the logs.

The third cause is data quality, and it is the only one that can hurt answers while leaving latency untouched. Vectors whose source document was deleted months ago are still retrievable and still cited in answers. A change of embedding model leaves two incompatible populations in one index if the backfill was partial. A change of dimension fails outright rather than degrading, because the dimension of a k-NN field and the vector(n) of a pgvector column are both fixed when the index is created. Duplicate chunks from a source connected twice crowd the top-k so one document supplies every result. Chunks whose text extraction failed embed as near-empty vectors that sit close to everything. None of this trips a metric. It is found by looking, which is why the looking has to be scheduled.

Getting the diagnosis wrong hurts in both directions. Adding OCUs or a bigger instance to an index-degradation problem delays the symptom by a few weeks at permanent extra cost and leaves recall exactly where it was. Booking a maintenance window to rebuild an index that is merely saturated uses up the window and returns the same latency.

What we’ll filter on

1. Which cause the evidence points at: saturation, index degradation, or data quality. Each has a distinct fingerprint and they can be present together.
2. Whether the signal already exists: does the store publish this metric, or does it have to be built and emitted as a custom one?
3. Whether the remedy runs online: can it happen against a live index, or does it need a rebuild, an alias swap, and a window?
4. Whether a re-embed is required: a rebuild reuses the vectors you have; a re-embed regenerates them and costs corpus-sized inference.
5. Whether it can be automated on a schedule, or whether it needs a person deciding each time.

The landscape

What each store publishes

OpenSearch Serverless. In the AWS/AOSS namespace a collection publishes SearchRequestLatency, SearchRequestErrors, IngestionDocumentErrors, SearchableDocuments, DeletedDocuments and 2xx/4xx/5xx counts. SearchOCU and IndexingOCU are reported for the account, or for the collection group the collection belongs to, and that is where the capacity ceiling sits: 10 OCUs each for indexing and search by default, adjustable up to 1,700. The latency metric carries minimum, maximum and average rather than percentiles, so a p99 has to come from your own tracing. Serverless does not expose cluster internals, so there is no JVM figure and no k-NN statistics API.

Provisioned OpenSearch domains. A domain publishes far more: CPUUtilization, JVMMemoryPressure and OldGenJVMMemoryPressure, SearchLatency, IndexingLatency, FreeStorageSpace, ClusterStatus.green, .yellow and .red, ThreadpoolSearchRejected, and the k-NN plug-in’s own statistics. The one that predicts an index outgrowing its nodes is KNNGraphMemoryUsagePercentage, the native memory held by k-NN graphs as a percentage of knn.memory.circuit_breaker.limit. At 100% the breaker trips and new knn_vector indexing is rejected. Below that, KNNEvictionCount and KNNMissCount rising together mean graphs are being dropped from the cache and reloaded from disk on the next query, which is where the latency curve steepens.

Aurora PostgreSQL with pgvector. The instance publishes CPUUtilization, DatabaseConnections, FreeableMemory, ReadIOPS and BufferCacheHitRatio, and Performance Insights attributes load to individual statements, so the vector query is visible separately from everything else the database is doing. A falling cache hit ratio with rising read IOPS is the index no longer being served from memory. The vector-specific health lives in the catalogue rather than CloudWatch: dead-tuple counts in pg_stat_user_tables, the last_autovacuum timestamp on the embeddings table, and index size against table size for bloat. An ingestion job that slows from forty minutes to four hours with CPU at 80% is usually autovacuum falling behind the write rate, not a query problem at all.

The application side. Whatever the store reports, the number a user feels is retrieval latency measured in your own trace, separate from generation latency. Splitting the two in the application’s own observability is what lets you say retrieval tripled while generation stayed flat.

Index maintenance routines

Index maintenance belongs on a cadence rather than after a complaint, and what is available differs sharply by store.

On a provisioned OpenSearch domain, the first routine is a force merge with only_expunge_deletes, which reclaims the graph memory the deleted documents were holding and shortens traversals. An Index State Management policy can schedule it. The second is a _reindex into a freshly built index with parameters sized for the corpus as it now stands, published behind an alias so the swap is atomic and reversible.

OpenSearch Serverless offers neither. Neither _forcemerge nor _reindex appears in its supported API set, and Index State Management is not among its plug-ins; segment merging is the service’s job rather than yours. What you get instead is the DeletedDocuments metric to watch, and a rebuild that means creating a second index and re-ingesting into it. That makes how the index is split an operational decision as much as a performance one: a rebuild’s blast radius is one index.

On pgvector, REINDEX INDEX CONCURRENTLY rebuilds an HNSW or IVFFlat index without locking reads or writes on the table, at the cost of running longer and holding both copies while it runs. Autovacuum settings on a high-churn embeddings table usually need tightening from the defaults, because vacuum is what reclaims dead tuples and keeps the index from bloating. VACUUM ANALYZE after a bulk load keeps the planner honest about whether to use the vector index at all.

IVF has its own routine. The centroids are trained once on a sample, so retraining and rebuilding is what corrects for corpus drift, and there is no incremental version. It is also not available everywhere: Serverless vector collections run HNSW on faiss and support neither IVF nor IVFQ, so there the graph is the only structure to rebuild. HNSW has no training step, and no way to raise m in place, so its rebuild is unavoidable once the graph is undersized for the corpus.

Query-time levers

Some of the latency is in the query rather than the index, and those levers are per request: ef_search on an HNSW index, spelled hnsw.ef_search in pgvector, the number of probed lists on IVF, spelled ivfflat.probes, the value of k, and whether a metadata filter runs during traversal or after it. These are the knobs that trade recall against latency, and unlike a rebuild they take effect on the next query. A filter applied after a k-of-100 scan is a hundred vectors of work to return five. On pgvector that behaviour has a version attached: without hnsw.iterative_scan, added in pgvector 0.8.0, a selective filter runs after the HNSW scan and returns fewer rows than were asked for.

Data quality checks

The validation worth scheduling is four checks, none of them slow:

• Reconciliation. Compare the set of source document identifiers against the set of documents in the index and report both directions. Documents in the source and not the index are ingestion failures that left no error; documents in the index and not the source are orphans still being cited.
• Dimension and model assertion. Every vector in an index must have the same dimension and come from the same embedding model. Record the model identifier and dimension as metadata on write, then count distinct values. More than one means the index is mixed.
• Duplicate detection. Hash chunk text and count collisions. A source connected twice, or an ingestion job re-run without a delete, shows up here before it shows up as a monotonous top-k.
• Empty and degenerate chunks. Flag chunks below a length floor, or whose vector norm is an outlier. These are usually failed text extraction, and they behave as universal near-neighbours, which is one route to the wrong passage being retrieved.

A scheduled recall probe

None of the above measures recall, and recall is the thing that actually degraded. Build a probe: a fixed set of query-to-expected-chunk pairs, forty or so, curated once from real questions with the right passage identified by a human. Run it on a schedule against production, compute recall at k and mean reciprocal rank, and emit both as CloudWatch custom metrics with an alarm on the drop. That turns “the assistant feels vaguer” into a graph with a date on it, and it is the only signal here that separates an index that is slow from an index that is wrong. Keep the pairs stable and version them alongside the ingestion pipeline, since a probe whose expected chunks have been re-chunked underneath it reports a fall that never happened.

Evaluation

Side by side

Lever	Fixes latency	Restores recall	Runs online	Schedulable	Signal that calls for it
Add capacity (OCUs, instance, replicas)	✓	✗	✓	✗	OCU or CPU at ceiling, JVM pressure, connections saturated
Tune ef_search, probes, k, filter order	✓	✓	✓	✗	Latency high with headroom to spare; post-filter discarding most results
Force merge to expunge deletes (domain only)	✓	✓	✓	✓	DeletedDocuments climbing against SearchableDocuments
Reindex behind an alias	✓	✓	✗	✓	Corpus several times the size the graph was built for
Retrain IVF centroids and rebuild	✓	✓	✗	✓	Probe recall falling on an IVF index with a stale training sample
Re-embed the corpus	✗	✓	✗	✗	Embedding model or dimension changed; mixed populations in one index
Reconcile and repair the data	✗	✓	✓	✓	Orphans, duplicates, mixed dimensions, or degenerate chunks found

Adding capacity is the only lever that fixes latency and does nothing at all for recall, which makes it the wrong lever whenever the probe has moved. Re-embedding is the only lever that fixes recall and does nothing for latency, and it is the most expensive one on the list, so it needs a specific trigger rather than a suspicion.

Reading saturation first is tempting because those metrics are already there, but a saturated store and a degraded index produce the same latency curve, and only the probe separates them.

The solution

Three standing pieces make this operable: an alarm set, a maintenance cadence, and a probe.

The alarm set

On an OpenSearch Serverless collection, alarm on SearchOCU and IndexingOCU crossing about 70% of the maximum configured for the account or collection group, on SearchRequestErrors and the 4xx and 5xx counts, and on IngestionDocumentErrors above zero. SearchRequestLatency arrives as average and maximum, so use it for the trend and take the p99 from the application trace. Set the maximum capacity deliberately rather than leaving it at the default 10 OCUs each, because that ceiling is both a spend limit and the number your utilisation alarm is measured against.

On a provisioned OpenSearch domain, AWS publishes a recommended set and it is the place to start: ClusterStatus.red at 1, ClusterStatus.yellow sustained, CPUUtilization at or above 80% for fifteen minutes, JVMMemoryPressure at 95% with OldGenJVMMemoryPressure at 80%, FreeStorageSpace below a quarter of each node’s storage, ClusterIndexWritesBlocked at 1, and any increase in ThreadpoolSearchRejected. Add KNNGraphMemoryUsagePercentage, since a tripped circuit breaker stops new vector indexing outright.

On Aurora with pgvector, alarm on CPUUtilization, DatabaseConnections against the connection limit, FreeableMemory falling toward the size of the index, and BufferCacheHitRatio dropping, which is the index no longer being served from memory. Add two custom metrics from the catalogue: dead tuples on the embeddings table, and hours since the last autovacuum on it. The four-hour ingestion in this scenario is that second metric, and the fix is autovacuum tuning on that one table, not a bigger instance.

The maintenance cadence

Weekly, run the four validation checks: reconciliation both directions, distinct dimension and model count, duplicate hashes, degenerate chunk count. Publish the counts as metrics so a rise is visible before it is a complaint.

Weekly or nightly, depending on delete volume, force merge on a domain, or confirm autovacuum has kept up on pgvector. On Serverless there is no equivalent to run, so DeletedDocuments becomes the trigger for the next rebuild instead. Either way, something has to stop 900,000 deleted documents sitting in the graph for a year.

Quarterly, or whenever the corpus grows past roughly twice the size the index was built for, rebuild. Build the new index with parameters chosen for the corpus as it now stands, load it, run the probe against both, and cut over only when the new index measures better. An IVF index retrains its centroids as part of that rebuild; an HNSW index gets a larger m and a build-time ef_construction to match. Doing it on a cadence, with the probe as the acceptance test, is what makes it a routine rather than an incident.

The rule about re-embedding

Whenever the embedding model or the dimension changes, a re-embed is required and a reindex is not sufficient, because vectors from two models are not comparable even at identical dimension, and distances computed across them are meaningless. Run it as a build into a new index, and keep the old one queryable until the probe passes on the new one. Tag every vector with its model identifier and dimension at write time, so a mixed state is detectable rather than inferred. With a Bedrock Knowledge Base the cutover is coarser than an alias swap: knowledgeBaseConfiguration and storageConfiguration cannot be changed after creation, so a new embedding model or a new vector index means a second knowledge base and a new identifier in the application. This is also the reason the store you chose matters operationally: the cost of a full re-embed and swap is a property of the store as much as of the corpus.

Worked example

Take the two teams in turn.

The OpenSearch assistant

The probe goes in first, forty query-to-chunk pairs curated from real support questions. The first run reports recall at 5 of 0.79. There is no history to compare it against, so the team builds a small index from the corpus as it stood a year ago and scores 0.93 on the same pairs. The drift is now a number.

Recall has fallen, so the flow goes down the left. Model and dimension: every vector carries the same model tag and 1,024 dimensions, so no re-embed. Reconciliation is uglier: 912,000 orphaned documents whose source was archived, no dimension mismatches, 4,100 duplicate chunk hashes from a source connected twice in the spring, and 800 chunks under the length floor. The short ones are a batch of scanned PDFs whose text extraction failed.

That is a data quality answer and an index answer together. The repair deletes the orphans and the duplicates, and re-extracts the scanned batch. On Serverless the deletes only push DeletedDocuments higher, with no force merge to reclaim them, so the repair runs straight into the rebuild. A new index goes up with m sized for 11 million live chunks rather than 2 million, is loaded from source, and is probed behind a second knowledge base. The application switches to it when it measures 0.94 at a p99 of 130 milliseconds. No extra OCUs were added, and the utilisation alarm never fired because utilisation was never the fault.

The pgvector assistant

Their probe is flat and their p99 is fine, so the flow goes right at the first gate. Saturation: CPU at 80% during ingestion, connections comfortable, freeable memory falling through the night and recovering by morning. The catalogue tells the rest of it. Dead tuples on the embeddings table are in the millions and autovacuum last completed on that table two days ago, because the nightly upsert churns more rows than the default thresholds trigger on.

The remedy is autovacuum tuning on that one table, a lower scale factor and more workers, plus a REINDEX INDEX CONCURRENTLY on the vector index to reclaim the bloat that has already accumulated. Ingestion returns to under an hour. Nothing about the index parameters was wrong, and a bigger instance would have hidden the problem for another quarter and then met it again.

What’s worth remembering

1. An approximate index degrades without an error, so recall has to be measured on a schedule with a fixed probe set and emitted as a custom metric; nothing the store publishes reports that the neighbours got worse.
2. Slow-and-stale retrieval has three causes with disjoint remedies: capacity saturation, index degradation, and data quality. Diagnose before you choose, because capacity fixes latency only and re-embedding fixes recall only.
3. Deleted documents keep occupying graph memory and query time until a merge reclaims them. A domain can force merge on a schedule and pgvector has autovacuum; OpenSearch Serverless has neither, so there a rising DeletedDocuments count is a rebuild trigger.
4. An HNSW m and an IVF centroid set are chosen for a corpus size and shape; when the corpus outgrows them the only remedy is a rebuild, cut over behind an alias or a second knowledge base with the probe as the acceptance test.
5. Whenever the embedding model or the dimension changes, re-embed rather than reindex, and tag every vector with its model and dimension so a mixed index is detectable. A Bedrock Knowledge Base cannot be repointed, so that cutover is a second knowledge base.
6. Weekly reconciliation in both directions, source against index, catches orphans and ingestion gaps that no latency or utilisation metric will ever surface.