Choosing an Embedding Dimension and Its Storage Cost

The situation

A knowledge-base team is building retrieval-augmented answering on Amazon Bedrock. They have roughly ten million passages today, growing toward maybe forty million as more document sources come online. Each passage is embedded and stored in a vector index that a retrieval step queries on every question, and the answer quality depends on the top handful of passages coming back relevant.

They started on Amazon Titan Text Embeddings v2 at its default 1024 dimensions because that was the number in the first tutorial they read. The index is already tens of gigabytes, the managed vector store bill is the fastest-growing line in the account, and query latency at the p99 is starting to be noticeable in the chat experience. Someone asked the obvious question: Titan v2 will emit 512 or 256 dimensions instead, so could they halve or quarter the footprint, and what would it cost them in answer quality?

Nobody wants to re-embed forty million passages twice by trial and error, and nobody wants to ship a smaller index that quietly returns worse passages. The decision underneath is the same one every embedding project reaches: how many dimensions, given this corpus size, this quality bar, and this storage and latency budget.

What actually matters

An embedding dimension is how many numbers each vector carries, and every one of those numbers is stored and compared for every vector in the index. That is the whole cost story in one sentence: dimension multiplies against the number of vectors and against the bytes each number takes. Doubling the dimension roughly doubles the raw vector bytes, the memory the index holds, and the arithmetic each similarity comparison performs. On a ten-passage toy index none of this matters; on tens of millions of passages it is the dominant term in the bill.

More dimensions can represent more nuance. A higher-dimensional space has more room to separate subtly different meanings, so on hard, semantically dense corpora a larger embedding often retrieves better. The catch is that the relationship is not linear and it is not guaranteed. Past the point where the extra dimensions stop capturing distinctions your queries actually depend on, you are paying full price in storage and latency for retrieval quality that has flattened out. Bigger is a hypothesis to test, not a rule to assume.

A lower dimension is cheaper on every axis at once: less storage, a smaller in-memory index, faster distance computations, and often lower latency. It buys that with some loss of fidelity, and how much loss depends entirely on your data and your quality bar. Some corpora barely notice the drop from 1024 to 512; others fall off a cliff. The only way to know which one you have is to measure retrieval quality at each dimension against a set of real queries with known-good answers, using something like recall@kRecall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. or a downstream answer-quality score, rather than reasoning about it in the abstract.

Titan Text Embeddings v2 makes this trade explicit by supporting configurable output dimensions of 1024, 512, and 256, with 1024 the default. The model is trained so the shorter vectors still carry most of the useful signal, which is what makes stepping down a genuine option rather than a straight truncation of quality. Not every model offers this; the older Titan Text Embeddings v1 emits a fixed 1536, and Cohere’s Bedrock embeddings have their own fixed sizes. Configurability is a property of the specific model, so it is worth confirming before you plan around it.

Two constraints sit underneath all of this and break the index silently if you get them wrong. The distance metric and the model have to match what the index was built for. An index configured for cosine similarity expects vectors compared by angle; one configured for Euclidean or dot-product expects something else, and querying with the wrong metric returns plausible-looking nonsense. The same holds for the model and dimension: every vector in one index must come from the same model at the same dimension, because vectors from different models or different sizes do not live in a comparable space. Re-embedding at a new dimension means rebuilding the index, not mixing sizes in place.

Normalisation is the quiet companion to the metric. Titan v2 can return normalised vectors (unit length), which is the setting you usually want, because with normalised vectors cosine similarity and dot product rank results identically, and cosine is the metric most retrieval setups assume. Turn normalisation off and dot-product magnitudes start reflecting vector length rather than just direction, which changes the ranking. The rule is to normalise consistently and pair it with a metric that matches, across both indexing and querying.

What we’ll filter on

1. Corpus size, how many vectors the index holds now and at projected growth, since dimension multiplies against every one of them.
2. Retrieval-quality bar, the recall@k or answer-quality floor the application needs, measured on real queries rather than assumed.
3. Storage and memory footprint, the raw vector bytes plus index overhead the budget can carry.
4. Query latency, whether smaller vectors and a smaller index keep p99 inside the experience’s budget.
5. Model support, whether the chosen model actually offers configurable dimensions, and which sizes.
6. Metric and normalisation fit, whether the distance metric and normalisation match across the model, the index, and the query path.

The dimensionality landscape

1024 dimensions (Titan v2 default). The most nuance the model offers and the safest starting point for quality, because you are not asking it to compress anything. It is also the most expensive on every axis: largest vectors, largest index, most arithmetic per comparison. Sensible as the baseline you measure everything else against, and the right resting place if your data genuinely needs the fidelity and the budget carries it.

512 dimensions. Half the raw storage and roughly half the per-comparison work of 1024, for a fidelity cost that is often small on well-behaved corpora. This is frequently the sweet spot for large indexes: the point where the footprint saving is real and the recall drop, if any, sits inside the quality bar. It is the first alternative worth measuring against the 1024 baseline.

256 dimensions. A quarter of the 1024 storage and the fastest to search. The fidelity cost is larger and more corpus-dependent, so it shines when the corpus is huge and cost-sensitive, the queries are relatively distinguishable, and measurement confirms recall still clears the bar. Attractive precisely where storage dominates, provided the quality holds.

Titan v1 at fixed 1536, and other fixed-size models. Not a dial at all. The dimension comes with the model, so the only way to change footprint is to change models and re-embed. Worth naming because a fixed-size model removes this lever entirely; if footprint tuning matters, prefer a model that offers configurable dimensions.

Quantisation, an orthogonal lever. Separate from dimension is how many bytes each number takes. Vectors are commonly stored as 32-bit floats (4 bytes each), and many vector stores can hold them as 8-bit integers or even binary instead, cutting the bytes-per-value several-fold with its own quality trade to measure. Dimension and quantisation stack: you can reduce the count of numbers and the size of each number independently. Keep them as two separate experiments so you can attribute the quality change correctly.

Side by side

Option	Relative storage	Search speed	Retrieval fidelity	Tunable	Best when
Titan v2, 1024	Highest (baseline)	Slowest	Highest	✓	Dense corpus, quality-led, budget carries it
Titan v2, 512	~½ of 1024	Faster	Usually close to 1024	✓	Large index, footprint matters, small recall cost
Titan v2, 256	~¼ of 1024	Fastest	Lower, corpus-dependent	✓	Huge, cost-sensitive index, quality still clears bar
Titan v1, 1536	Highest, fixed	Slowest	High	✗	Legacy path; no dimension lever
Quantised storage (int8 / binary)	Cuts bytes-per-value	Faster	Trade to measure	✓	Stacks on any dimension to cut footprint further

Reading the table against the ten-to-forty-million-passage index: 1024 is the quality baseline to beat, 512 is the first serious candidate for halving the footprint, 256 is on the table if measurement says the corpus tolerates it, and quantisation is a second, independent saving to layer on once the dimension is settled.

The picks in depth

Start by doing the storage arithmetic, because it decides how much the choice is worth. The raw vector footprint is vectors times dimension times bytes-per-value. At ten million passages, 1024 dimensions, and 4-byte floats that is 10,000,000 times 1024 times 4, about 41 GB of raw vectors; at 512 it is roughly 20 GB, and at 256 roughly 10 GB. Project to forty million and those become around 164, 82, and 41 GB. On top of the raw vectors, the index structure itself carries overhead. A graph-based index such as HNSW stores neighbour links per vector, which can add a meaningful fraction again, and that overhead also scales with the number of vectors. The vectors dominate a large index’s footprint, which is exactly why dimension, the multiplier on those vectors, is the highest-leverage number to get right.

With the money at stake sized, the pick is a measurement, not a guess. Embed a representative sample at 1024, 512, and 256, build an index for each, and run the same query set with known-relevant passages through all three, scoring recall@k or the downstream answer quality. If 512 holds the quality bar, you have halved the largest line in the storage and memory bill for little or no loss, which on a forty-million-vector index is a large, permanent saving. If 256 also holds, take it. If quality falls off between 1024 and 512, you have learned your corpus needs the fidelity, and the money is buying something real. The point is that the data answers the question; assuming bigger is better leaves savings on the table, and assuming smaller is fine ships worse answers.

Whatever dimension you land on, get the metric and normalisation right once and consistently. Pick cosine similarity unless you have a specific reason not to, ask Titan v2 for normalised vectors, and build the index with the matching metric. Then keep the query path identical: same model, same dimension, same normalisation, same metric on both the indexing and the querying side. A mismatch here does not throw an error; it just quietly retrieves worse passages, which is the hardest kind of bug to notice in a retrieval system because the output still looks like an answer. Re-embedding at a new dimension is a full index rebuild, so plan the cutover rather than trying to change size in place.

A worked example: sizing the step from 1024 to 512

Take the index at its projected forty million passages. At 1024 dimensions and 4-byte floats the raw vectors are 40,000,000 times 1024 times 4, about 164 GB, before index overhead; the HNSW graph on top might add roughly a third again, pushing the working set past 200 GB and well into the range where it stops fitting comfortably in memory on a modest node. Query latency at the p99 is climbing because larger vectors mean more arithmetic per comparison and a bigger structure to traverse.

The team samples two million passages, embeds them at 1024, 512, and 256, and measures recall@10 against a curated set of a few hundred real questions with hand-checked relevant passages. Recall@10 comes back at 0.94 for 1024, 0.93 for 512, and 0.88 for 256. The application’s floor is 0.90. That reading makes the decision for them: 512 costs a single point of recall and stays above the bar, while 256 drops below it on this corpus. They re-embed at 512 and rebuild the index. The raw vectors fall to about 82 GB, the working set comes back within a comfortable memory envelope, p99 latency eases, and the monthly storage line roughly halves, all for a recall cost the quality bar can absorb. Had the numbers landed differently, with 512 falling to 0.87, the same experiment would have told them to stay at 1024 and instead look at quantisation for savings. Either way the measurement, not the assumption, chose the number.

What’s worth remembering

1. Embedding dimension multiplies against every vector in the index, so on a large corpus it is the dominant term in storage, memory, and per-comparison search cost.
2. A higher dimension can capture more nuance, but the quality gain flattens and is not guaranteed; past a point you pay full cost for fidelity your queries never use.
3. A lower dimension is cheaper and faster on every axis at once, at a fidelity cost that depends entirely on your data and your quality bar.
4. Amazon Titan Text Embeddings v2 supports configurable output dimensions of 1024, 512, and 256, with 1024 the default; configurability is a property of the model, not a given.
5. Size the money first with vectors times dimension times bytes-per-value; ten million 1024-d float vectors are about 41 GB of raw vectors before index overhead.
6. Measure recall on your own data at each candidate dimension against real queries; do not assume bigger is better or smaller is fine.
7. The distance metric and the model must match what the index expects, and every vector in an index must share the same model and dimension.
8. Normalise consistently and pair it with a matching metric; with normalised vectors, cosine and dot product rank identically, and cosine is what most setups assume.
9. Quantisation (float32 to int8 or binary) cuts bytes-per-value independently of dimension, so treat it as a second, separately measured saving that stacks on the dimension choice.
10. Re-embedding at a new dimension is a full index rebuild, not an in-place change, so plan the cutover rather than mixing vector sizes in one index.