One Vector Index or Many

The situation

A retrieval layer behind a Bedrock assistant has been running on Amazon OpenSearch Service for eighteen months. It launched as a single vector index over four hundred thousand chunks and now holds roughly forty million across six business domains: product documentation, contracts, support history, engineering runbooks, marketing copy and finance policy. Eleven tenants share it, separated by a tenant identifier stored as metadata on every chunk and applied as a filter on every query.

Two numbers have gone bad. Retrieval p99 was around 180ms at launch and is now closer to 900ms against a 400ms budget, while the median has barely moved. A full rebuild, which is what a chunking or embedding-model change demands, takes a weekend and cannot be done in pieces. It is all forty million chunks or none of them.

The proposal on the table is to split the index. Nobody has said into what. Six domain indexes, eleven tenant indexes, sixty-six of both crossed together, monthly partitions, and a routing layer that searches a small index first are all on the whiteboard. Their costs differ enormously.

What actually matters

Start with memory, because index count is a memory decision before it is anything else. An HNSW graph is searched in memory rather than streamed off disk. Every graph in the cluster has to be resident for its shard to answer a query at all. The total is a function of vector count, dimensionEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. and the graph’s own per-vector overhead, multiplied by the number of copies you keep. Splitting one index into six does not reduce that total by a byte. It adds to it. Each additional index carries cluster-state entries, at least one primary shard with its own Lucene segments, and a floor of working memory before it holds a single vector. Reaching for more indexes to fix a memory ceiling makes the ceiling lower.

What splitting does change is how much of the corpus a query has to touch. A query against one index fans out to every primary shard, and the coordinating node waits for the slowest to come back. Tail latency is therefore set by the worst shard rather than the average one, which is exactly the shape of a median that holds while p99 doubles. A metadata filter narrows the results, not the fan-out: every shard still runs a search and still has to be waited for. Splitting so that a typical query touches one shard instead of six is a latency lever. Splitting so that a typical query still touches everything, now through six coordinator round trips instead of one, is not.

The third thing topology decides is the blast radius of a rebuild. Changing the chunker, the embedding model or the dimension means recomputing every vector that the change touches. With one index that is one job, one cutover and one rollback. A bad decision about one domain becomes a corpus-wide outage. With six it is six jobs that can run, cut over and be reverted independently. Refresh cadence pulls the same way: support history changes hourly and contracts change monthly, and a single index forces the whole corpus onto the schedule of its most volatile part.

Then isolation, which is the one that stops being a performance question and becomes a compliance question. A tenant filter is a boundary the application enforces, and it is correct only for as long as every query path in every service remembers to apply it. A separate index is a boundary the storage layer enforces, and, more usefully, one that can be deleted. Removing a tenant from a shared HNSW index means deleting documents and then waiting for segment merges before the vectors leave the graph. Removing a tenant’s index is a single call with a defensible answer attached. That has to be weighed against the fixed cost, because eleven tenant indexes and sixty-six tenant-by-domain indexes are very different propositions.

What we’ll filter on

1. Fan-out: how many shards does one query touch, and does the topology genuinely narrow that or only rearrange it?
2. Fixed cost per index: what does each extra index cost in k-NN graph memory, shard overhead and cluster state before it holds any data?
3. Rebuild blast radius: when the embedding model or chunking changes, what is the smallest unit that can be rebuilt and rolled back on its own?
4. Filter selectivity: does a query arrive carrying an attribute selective enough to route on, or does it have to search everything?
5. Isolation and deletion: can a tenant’s vectors be separated and hard-deleted at the storage layer rather than by application convention?
6. Operational load: how many indexes, ingestion jobs, aliases and cutovers does somebody have to run every week?

The landscape

Three levers decide retrieval performance at this scale, and they combine rather than compete: how an index is sharded, how many indexes the corpus is spread across, and whether a small routing index sits in front of the big ones. At forty million chunks, the combination of the three settles more than any graph parameter tuned afterwards.

One index with metadata filters

Today’s arrangement, and the one to beat. Every chunk lives in one index with tenant_id and domain as keyword fields, and queries run filtered k-NN. Modern OpenSearch applies that filter during graph traversal rather than after it, and falls back to an exact scan when the filtered set is small enough that scanning beats walking a graph. A highly selective filter can therefore be quicker than an unfiltered search, not slower. Every one of the forty million vectors sits in one graph family, scores come from one model under one distance metric, and a cross-domain question is answered in a single hop.

The sharding decisions live here. Shard size is the lever, and the rule of thumb for search workloads puts a shard in the tens of gigabytes; for a vector index the binding constraint is usually the graph memory a shard’s vectors demand rather than the bytes on disk. Primary shard count is fixed when the index is created, which makes it a one-shot capacity decision sized for where the index is going rather than where it is. Changing it later means a reindex, or a split or shrink into a new index, which is the same weekend either way. Replicas are a separate lever: in OpenSearch a replica shard serves searches, so replica count is how you add query throughput, at the cost of a second resident copy of every graph it holds.

One index per domain

Six indexes, each sized for its own content. A query that knows its domain is sent to one of them and touches only that index’s shards. A query that does not know its domain either goes through a routing step that works it out, or fans out across a wildcard alias covering all six and merges what comes back.

The two variants behave very differently. Routing narrows the fan-out and is the version that helps p99. Alias fan-out searches everything anyway, so it gives you independent rebuilds and per-domain cadence without the latency win, and it introduces a merge problem covered below.

One index per tenant

Eleven indexes, or sixty-six if you cross tenant with domain. Every tenant’s data is physically separate, deletion is a single operation, and a per-index access policy can back up the application’s filter with something the cluster enforces. The cost is that tenants are never evenly sized. The two largest here hold two thirds of the corpus; the smallest holds forty thousand chunks, a shard of a few hundred megabytes carrying the same fixed overhead as one fifty times its size.

Time-partitioned indexes behind an alias

Monthly or quarterly indexes with an alias spanning them, borrowed from log-analytics practice. It works when recency dominates the query mix. Most searches point at an alias covering the last two partitions, and old data ages out by dropping an index rather than by deleting documents. It works badly for a corpus like contracts, where a five-year-old document is exactly as relevant as yesterday’s.

A two-stage hierarchical design

The hierarchical indexing technique. A small routing index holds one vector per domain, or one per document cluster, built from summaries rather than raw chunks. The query is searched against that routing index first, which is tiny and answers in single-digit milliseconds, and the top one or two results decide which domain index gets the real search. A forty-million-vector fan-out becomes a six-vector search followed by a search over one domain. The trade is a hard dependency on the router being right. A routing miss returns a confident answer from the wrong corpus, and nothing downstream will notice.

Evaluation

Side by side

Read the first two columns together and the trade sits there in the open: the only topology that costs no extra memory is the one that never narrows the fan-out, and every topology that narrows the fan-out pays a fixed cost per index to do it. So “we are running out of memory, let us split the index” is the wrong reflex, and “our p99 is set by shards we did not need to search” is a good reason to split.

The per-tenant row looks like the strongest one, with five ticks. It is also the row where the operational column hides the most. Eleven indexes is manageable; the sixty-six you get from crossing tenant with domain give you a cluster whose shard count and cluster-state churn become their own performance problem.

Where a split is justified

The ordering carries the argument. Memory and rebuild time come first because they are measurable this afternoon and they are the only reasons the split is unavoidable. Isolation comes second because it is the one requirement no amount of query tuning will satisfy. Recency and routability come last because they are optimisations, and an optimisation applied to a topology nobody needed is more indexes to run.

The solution

Stay on one index with metadata filters until either k-NN graph memory or a tenant-deletion requirement forces the split, and do the memory arithmetic before agreeing that it has. Most p99 drift at this size is a shard-sizing problem rather than a topology problem. Re-sharding the single index into shards of the right size, or dropping the embedding dimension so that every graph shrinks together, fixes it without adding a single index to operate. The same goes for the choice between HNSWHNSWA graph-based vector index that walks neighbour links to find close vectors fast, at the cost of extra memory per vector. and IVFIVFA vector index that clusters vectors up front and searches only the nearest clusters – cheaper memory than a graph index, more tuning., which is a per-index memory-versus-recall trade-off you should have settled before topology enters the conversation.

When a split is forced, split by domain rather than by tenant, and split because the domains want different things rather than because there are six of them. Two conditions justify a domain index on their own. The domain needs a different embedding model, which makes a shared index impossible rather than awkward. Or the domain has a refresh cadence that a shared rebuild schedule cannot serve. Support history re-embedded hourly next to contracts rebuilt monthly is the textbook case, and the freshness machinery gets simpler on both sides once they are separate. Size each domain index for its own content instead of copying a shard count across all six, because uniform sharding over uneven domains produces both oversized and near-empty shards in one move.

Reserve per-tenant indexes for the handful of tenants that are genuinely large or genuinely contractually separate, and leave the rest in the shared index behind the tenant filter. Thousands of small tenant indexes is the classic way to make this worse. Cluster state grows with every index, each shard carries fixed overhead whether it holds four hundred thousand vectors or four thousand, and a cluster busy with shard bookkeeping runs slower across the board than the one index you started with. Eleven tenants split into sixty-six indexes crossed with domain is the same mistake at smaller scale.

Whatever the topology, run every cutover through an index alias. Applications and knowledge bases point at the alias, never at the concrete index. A rebuild writes into a fresh index alongside the live one and gets validated against a held-out query set while the old index is still serving. It goes live as a single atomic alias swap, not a migration with a window in it. The old index stays until confidence is earned, and rollback is the same swap in reverse:

{ "actions": [
  { "remove": { "index": "kb-support-v3", "alias": "kb-support" } },
  { "add":    { "index": "kb-support-v4", "alias": "kb-support" } }
] }

Three gotchas will bite in order. First, Amazon OpenSearch Serverless removes most of this calculation: a vector collection has no shard settings to tune, capacity is billed in OpenSearch Compute Units with a floor per collection, so the cost of an extra index becomes an OCU question rather than a heap-and-graph question. That changes the shape of the answer without changing the reasoning, and it makes a scatter of small collections expensive in a way small indexes on a provisioned domain are not. Second, a Bedrock Knowledge Base owns exactly one vector index. A six-domain split therefore means six knowledge bases, six data source configurations, six ingestion jobs, and six Retrieve calls whenever a question crosses domains. Weigh that against the managed ingestion you get for free today. Third, and this is the one that fails silently, similarity scores are comparable across indexes only when every index uses the same embedding model, the same dimension and the same distance metric. Merging fan-out results by score breaks the moment one domain moves to a different model, which is one of the main reasons to split in the first place. Where domains genuinely differ, route to one index rather than fanning out. Otherwise re-rank the merged candidate set with a single reranking model, so the final ordering comes from one scorer.

Worked example

The memory floor

Forty million chunks embedded at 1024 dimensions in float32, indexed with HNSW at m = 16. OpenSearch documents graph memory as roughly 1.1 * (4 * dimensions + 8 * m) bytes per vector, so:

1.1 * (4 * 1024 + 8 * 16) = 1.1 * 4224 = 4,646 bytes per vector
40,000,000 * 4,646          = ~186 GB of primary graph
with one replica            = ~372 GB resident across the cluster

The k-NN plugin’s circuit breaker defaults to half of a node’s memory, so 372 GB of graph wants around 750 GB of data-node RAM. That is six r6g.4xlarge nodes with nothing spare, or eight with room to breathe. Put that number on the whiteboard before anyone draws a topology. At 186 GB of primary data, shards in the tens of gigabytes put the index at five or six primaries. That is also how many shards every single query currently waits on.

Note what a topology change does to that arithmetic: nothing. Six domain indexes hold the same forty million vectors and need the same 372 GB. Dropping to 512 dimensions takes the primary graph to about 96 GB and halves the cluster. That is why the dimension decision outranks the index-count decision when memory is the binding constraint.

What a domain split changes

The six domains are wildly uneven: support history at 22M chunks, product documentation at 9M, and the remaining four sharing the last 9M with contracts at 1.2M. Sized individually that is four primaries for support history, two for product documentation, and one each for the rest: ten primaries against the six you have now. A routed query hits four shards for a support question and one shard for a contracts question, instead of six for both, and the contracts rebuild is a twenty-minute job rather than a share of the weekend.

The cost is that a question spanning support history and product documentation now needs two searches and an application-side merge, sound only while both indexes run the same embedding model. The day product documentation moves to a domain-tuned model, that merge has to become a rerank.

Why sixty-six indexes costs more than it saves

Crossing eleven tenants with six domains gives sixty-six indexes. Even at one primary and one replica each, that is 132 shards, average population 600,000 vectors, average graph about 2.8 GB. Two thirds of them are under 200,000 vectors: a shard well under a gigabyte carrying the full fixed overhead of one. Shard-count guidance for a cluster this size is measured against heap, not against how tidy the naming scheme looks. You have added coordination, cluster-state churn and sixty-six ingestion pipelines, and the resident graph memory has not moved at all. The two tenants with a deletion clause get their own indexes; the other nine stay behind the filter.

What’s worth remembering

1. Splitting an index never reduces total k-NN graph memory and adds fixed overhead per index, so memory pressure is answered by dimension, index type or more nodes.
2. Tail latency is set by the slowest shard a query touches, and a metadata filter narrows results without narrowing the fan-out, so only a topology that routes queries to fewer shards moves p99.
3. Split by domain when domains need different embedding models or different refresh cadences, and size each domain index for its own content instead of copying one shard count across all of them.
4. Primary shard count is fixed at index creation and replicas serve searches, so shard count is a one-shot capacity decision and each replica costs a full extra copy of every graph.
5. Reserve per-tenant indexes for large or contractually separated tenants; thousands of tiny indexes turns a retrieval problem into a cluster-management problem.
6. Route every cutover through an index alias so a rebuild is an atomic swap with a swap-back available, and never merge scores across indexes that differ in embedding model, dimension or distance metric.