One Vector Index or Many

The situation

A retrieval layer behind a Bedrock assistant has been running on Amazon OpenSearch Service for eighteen months. It launched as a single vector index over four hundred thousand chunks and now holds roughly forty million across six business domains: product documentation, contracts, support history, engineering runbooks, marketing copy and finance policy. Eleven tenants share it, separated by a tenant identifier stored as metadata on every chunk and applied as a filter on every query.

Two numbers have gone bad. Retrieval p99 was around 180ms at launch and is now closer to 900ms against a 400ms budget, while the median has barely moved. A full rebuild, which is what a chunking or embedding-model change demands, takes a weekend and cannot be done in pieces. It is all forty million chunks or none of them.

The proposal on the table is to split the index. Nobody has said into what. Six domain indexes, eleven tenant indexes, sixty-six of both crossed together, monthly partitions, and a routing layer that searches a small index first are all on the whiteboard. The differences between them run to whole nodes and whole weekends.

What actually matters

Start with memory, because index count is a memory decision before it is anything else. An HNSW graph is searched in memory rather than streamed off disk. Every graph in the cluster has to be resident for its shard to answer a query at all. The total is a function of vector count, dimensionEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. and the graph’s own per-vector overhead, multiplied by the number of copies you keep. Splitting one index into six does not reduce that total by a byte. It adds to it. Each additional index carries cluster-state entries, at least one primary shard with its own Lucene segments, and a floor of working memory before it holds a single vector. Reaching for more indexes to fix a memory ceiling makes the ceiling lower.

What splitting does change is how much of the corpus a query has to touch. A query against one index fans out to every primary shard, and the coordinating node waits for the slowest to come back. Tail latency is therefore set by the worst shard rather than the average one, which is exactly the shape of a median that holds while p99 doubles. A metadata filter narrows the results, not the fan-out: every shard still runs a search and still has to be waited for. Splitting so that a typical query touches one shard instead of six is a latency lever. Splitting so that a typical query still touches everything, now through six coordinator round trips instead of one, is not.

The third thing topology decides is the blast radius of a rebuild. Changing the chunker, the embedding model or the dimension means recomputing every vector that the change touches. With one index that is one job, one cutover and one rollback. A bad decision about one domain becomes a corpus-wide outage. With six it is six jobs that can run, cut over and be reverted independently. Refresh cadence pulls the same way: support history changes hourly and contracts change monthly, and a single index forces the whole corpus onto the schedule of its most volatile part.

Then isolation, which is the one that stops being a performance question and becomes a compliance question. A tenant filter is a boundary the application enforces, and it is correct only for as long as every query path in every service remembers to apply it. A separate index is a boundary the storage layer enforces, and, more usefully, one that can be deleted. Removing a tenant from a shared HNSW index means deleting documents and then waiting for segment merges before the vectors leave the graph. Removing a tenant’s index is a single call with a defensible answer attached. That has to be weighed against the fixed overhead, because eleven tenant indexes and sixty-six tenant-by-domain indexes are very different propositions.

What we’ll filter on

1. Fan-out: how many shards does one query touch, and does the topology genuinely narrow that or only rearrange it?
2. Fixed overhead per index: what does each extra index add in k-NN graph memory, shard overhead and cluster state before it holds any data?
3. Rebuild blast radius: when the embedding model or chunking changes, what is the smallest unit that can be rebuilt and rolled back on its own?
4. Filter selectivity: does a query arrive carrying an attribute selective enough to route on, or does it have to search everything?
5. Isolation and deletion: can a tenant’s vectors be separated and hard-deleted at the storage layer rather than by application convention?
6. Operational load: how many indexes, ingestion jobs, aliases and cutovers does somebody have to run every week?

The landscape

Three levers decide retrieval performance at this scale, and they combine rather than compete: how an index is sharded, how many indexes the corpus is spread across, and whether a small routing index sits in front of the big ones. At forty million chunks, the combination of the three settles more than any graph parameter tuned afterwards.

One index with metadata filters

Today’s arrangement, and the one to beat. Every chunk lives in one index with tenant_id and domain as keyword fields, and queries run filtered k-NN. OpenSearch applies that filter during graph traversal rather than after it: Lucene HNSW since 2.4, Faiss HNSW since 2.9, Faiss IVF since 2.10. When the filtered set falls below knn.advanced.filtered_exact_search_threshold, it runs an exact pre-filtered search instead of walking the graph. A highly selective filter can therefore be quicker than an unfiltered search, not slower. Every one of the forty million vectors sits in one graph family, scores come from one model under one distance metric, and a cross-domain question is answered in a single hop.

The sharding decisions live here. Shard size is the lever. AWS puts a shard between 10 and 30 GiB for search workloads and 30 to 50 GiB for write-heavy ones, but for a vector index the binding constraint is usually the graph memory a shard’s vectors demand rather than the bytes on disk. Primary shard count is fixed when the index is created, which makes it a one-shot capacity decision sized for where the index is going rather than where it is. Changing it later means a reindex, or a split or shrink into a new index, which is the same weekend either way. Replicas are a separate lever. A search request is routed to either the primary or a replica for each shard in the index, so replica count is how you add query throughput, and each replica holds a second resident copy of every graph.

One index per domain

Six indexes, each sized for its own content. A query that arrives carrying its domain is sent to one of them and touches only that index’s shards. A query that arrives without one either goes through a routing step that works it out, or fans out across a wildcard alias covering all six and merges what comes back.

The two variants behave very differently. Routing narrows the fan-out and is the version that helps p99. Alias fan-out searches everything anyway, so it gives you independent rebuilds and per-domain cadence without improving p99, and it introduces a merge problem covered below.

One index per tenant

Eleven indexes, or sixty-six if you cross tenant with domain. Every tenant’s data is physically separate, deletion is a single operation, and a per-index access policy can back up the application’s filter with something the cluster enforces. The difficulty is that tenants are never evenly sized. The two largest here hold two thirds of the corpus; the smallest holds forty thousand chunks, a shard of a few hundred megabytes carrying the same fixed overhead as one fifty times its size.

Time-partitioned indexes behind an alias

Monthly or quarterly indexes with an alias spanning them, borrowed from log-analytics practice. It works when recency dominates the query mix. Most searches point at an alias covering the last two partitions, and old data ages out by dropping an index rather than by deleting documents. It works badly for a corpus like contracts, where a five-year-old document is exactly as relevant as yesterday’s.

A two-stage hierarchical design

The hierarchical indexing technique. A small routing index holds one vector per domain, or one per document cluster, built from summaries rather than raw chunks. The query is searched against that routing index first, which is tiny and answers in single-digit milliseconds, and the top one or two results select the domain index for the real search. A forty-million-vector fan-out becomes a six-vector search followed by a search over one domain. The dependency on the router being right is absolute. A routing miss returns a well-formed answer drawn from the wrong corpus, and nothing downstream flags it.

Evaluation

Side by side

Read the first two columns together and the trade sits there in the open: the only topology that adds no memory is the one that never narrows the fan-out, and every topology that narrows the fan-out adds fixed memory per index to do it. So “we are running out of memory, let us split the index” is the wrong reflex, and “our p99 is set by shards we did not need to search” is a good reason to split.

The per-tenant row looks like the strongest one, with five ticks. The operational column is doing the most work in that row. Eleven indexes is manageable; the sixty-six you get from crossing tenant with domain give you a cluster whose shard count and cluster-state churn become their own performance problem.

Where a split is justified

The ordering carries the argument. Memory and rebuild time come first because they are measurable this afternoon and they are the only reasons the split is unavoidable. Isolation comes second because it is the one requirement no amount of query tuning will satisfy. Recency and routability come last because they are optimisations, and an optimisation applied to a topology nobody needed is more indexes to run.

The solution

Stay on one index with metadata filters until either k-NN graph memory or a tenant-deletion requirement forces the split, and do the memory arithmetic before agreeing that it has. Most p99 drift at this size is a shard-sizing problem rather than a topology problem. Re-sharding the single index into shards of the right size, or dropping the embedding dimension so that every graph shrinks together, fixes it without adding a single index to operate. The same goes for the choice between HNSWHNSWA graph-based vector index that walks neighbour links to find close vectors fast, at the cost of extra memory per vector. and IVFIVFA vector index that clusters vectors up front and searches only the nearest clusters – cheaper memory than a graph index, more tuning., which is a per-index memory-versus-recall trade-off you should have settled before topology enters the conversation.

When a split is forced, split by domain rather than by tenant, and split because the domains have different requirements rather than because there are six of them. Two conditions justify a domain index on their own. The domain needs a different embedding model, which makes a shared index impossible rather than awkward. Or the domain has a refresh cadence that a shared rebuild schedule cannot serve. Support history re-embedded hourly next to contracts rebuilt monthly is the textbook case, and the freshness machinery gets simpler on both sides once they are separate. Size each domain index for its own content instead of copying a shard count across all six, because uniform sharding over uneven domains produces both oversized and near-empty shards in one move.

Reserve per-tenant indexes for the handful of tenants that are genuinely large or genuinely contractually separate, and leave the rest in the shared index behind the tenant filter. Thousands of small tenant indexes is the classic way to make this worse. Cluster state grows with every index, each shard carries fixed overhead whether it holds four hundred thousand vectors or four thousand, and a cluster busy with shard bookkeeping runs slower across the board than the one index you started with. Eleven tenants split into sixty-six indexes crossed with domain is the same mistake at smaller scale.

Whatever the topology, run every cutover through an index alias. Applications and knowledge bases point at the alias, never at the concrete index. A rebuild writes into a fresh index alongside the live one and gets validated against a held-out query set while the old index is still serving. It goes live as a single atomic alias swap, not a migration with a window in it. The old index stays in place while the new one is under watch, and rollback is the same swap in reverse:

{ "actions": [
  { "remove": { "index": "kb-support-v3", "alias": "kb-support" } },
  { "add":    { "index": "kb-support-v4", "alias": "kb-support" } }
] }

Three gotchas, in the order they show up. First, Amazon OpenSearch Serverless removes most of this arithmetic. A vector search collection has no shard settings to tune, and capacity is measured in OpenSearch Compute Units of 6 GiB each, with minimum and maximum limits set per collection group rather than per collection or index. Collections in a group share OCUs and the minimum can be set to zero, so an extra index becomes an OCU question rather than a heap-and-graph one. The reasoning above survives the move; only the arithmetic changes. Second, a Bedrock knowledge base points at exactly one vector index. A six-domain split therefore means six knowledge bases, six data source configurations, six ingestion jobs, and six Retrieve calls whenever a question crosses domains. Weigh that against the managed ingestion a single knowledge base handles. Third, and this is the one that fails silently, similarity scores are comparable across indexes only when every index uses the same embedding model, the same dimension and the same distance metric. Merging fan-out results by score breaks the moment one domain moves to a different model, which is one of the main reasons to split in the first place. Where domains genuinely differ, route to one index rather than fanning out. Otherwise re-rank the merged candidate set with a single reranking model, so the final ordering comes from one scorer.

Worked example

The memory floor

Forty million chunks embedded at 1024 dimensions in float32, indexed with HNSW at m = 16. OpenSearch estimates graph memory at 1.1 * (4 * dimensions + 8 * m) bytes per vector, so:

1.1 * (4 * 1024 + 8 * 16) = 1.1 * 4224 = 4,646 bytes per vector
40,000,000 * 4,646          = ~173 GiB of primary graph
with one replica            = ~346 GiB resident across the cluster

Now the node count, and this is where the estimate usually goes wrong. OpenSearch Service gives half of an instance’s RAM to the Java heap, capped at 32 GiB, and the k-NN circuit breaker defaults to half of what remains. A node holds about a quarter of its RAM in graphs, not half. So 346 GiB of graph needs roughly 1.4 TiB of data-node RAM: eleven r7g.4xlarge.search nodes at 128 GiB each, 32 GiB of graph apiece, nothing spare. Put that number on the whiteboard before anyone draws a topology. At 173 GiB, the top of the 10 to 30 GiB search-workload shard range puts the index at six primaries. That is also how many shards every single query currently waits on.

Note what a topology change does to that arithmetic: nothing. Six domain indexes hold the same forty million vectors and need the same 346 GiB. Dropping to 512 dimensions takes the primary graph to about 89 GiB and halves the cluster. That is why the dimension decision outranks the index-count decision when memory is the binding constraint.

What a domain split changes

The six domains are wildly uneven: support history at 22M chunks, product documentation at 9M, and the remaining four sharing the last 9M with contracts at 1.2M. Sized individually that is four primaries for support history, two for product documentation, and one each for the rest: ten primaries against the six you have now. A routed query hits four shards for a support question and one shard for a contracts question, instead of six for both, and the contracts rebuild is a job measured in hours rather than a share of the weekend.

What that gives up is cross-domain retrieval in one hop. A question spanning support history and product documentation now needs two searches and an application-side merge, which is sound only while both indexes run the same embedding model. The day product documentation moves to a domain-tuned model, that merge has to become a rerank.

Why sixty-six indexes make it worse

Crossing eleven tenants with six domains gives sixty-six indexes. Even at one primary and one replica each, that is 132 shards, average population 600,000 vectors, average graph about 2.6 GiB. Two thirds of them are under 200,000 vectors: a shard well under a gigabyte carrying the full fixed overhead of one. You have added coordination, cluster-state churn and sixty-six ingestion pipelines, and the resident graph memory has not moved at all. The two tenants with a deletion clause get their own indexes; the other nine stay behind the filter.

What’s worth remembering

1. Splitting an index never reduces total k-NN graph memory and adds fixed overhead per index, so memory pressure is answered by dimension, index type or more nodes.
2. Tail latency is set by the slowest shard a query touches, and a metadata filter narrows results without narrowing the fan-out, so only a topology that routes queries to fewer shards moves p99.
3. Split by domain when domains need different embedding models or different refresh cadences, and size each domain index for its own content instead of copying one shard count across all of them.
4. Primary shard count is fixed at index creation and replicas serve searches, so shard count is a one-shot capacity decision and each replica holds a full extra copy of every graph.
5. Reserve per-tenant indexes for large or contractually separated tenants; thousands of tiny indexes turns a retrieval problem into a cluster-management problem.
6. Route every cutover through an index alias so a rebuild is an atomic swap with a swap-back available, and never merge scores across indexes that differ in embedding model, dimension or distance metric.