Choosing a Distance Metric for Embeddings

The situation

A team has built a retrieval-augmented feature on Amazon Bedrock. Documents are chunked, run through amazon.titan-embed-text-v2:0 to produce 1,024-dimension vectors, and stored in an OpenSearch k-NNNearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed. index; at query time the user’s question is embedded the same way and the store returns the nearest chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. to stuff into the prompt. It worked in the prototype. In production, retrieval quality is oddly mediocre: the top hits are often loosely related rather than on the nose, and the relevant chunk that a human can find in seconds sometimes sits at rank forty instead of rank one.

Nothing in the monitoring registers a fault. The index builds, queries return in single-digit milliseconds, there are no exceptions, and the embedding calls all succeed. When someone finally diffs the index settings against the model documentation, two settings turn out to disagree. The index was created without a space_type, so it took OpenSearch’s default of l2, while the embedding calls pass normalize: false, so the vectors come back at whatever length the model produces. Differences in vector length had been driving the ranking all along.

The underlying question is small and easy to get wrong: which distance metric should the vector store use, and how do you know it matches the model that produced the embeddings?

What actually matters

An embedding is a point in a few hundred or few thousand dimensions, and “similar” means “close” under some definition of distance. The definition is not free to choose after the fact. The model was trained against a specific notion of closeness, and the numbers it emits carry meaning under that notion. The metric is not a tuning knob you turn for better results; it is a property of the model, and the index has to match it.

The dividing line that decides the most is whether magnitude carries meaning. Cosine similarity looks only at the angle between two vectors and ignores how long they are, so a short vector and a long vector pointing the same way score identically. Dot product (inner product) multiplies angle and magnitude together, so a longer vector scores higher for the same direction. Euclidean distance measures the gap between the two points, which blends direction and magnitude into one number and is dominated by magnitude when the lengths vary. For most text embedding models the meaning sits in the direction. That is why cosine is the common default for text.

Normalisation is what turns this from trivia into a real failure mode. A vector is normalised when it is scaled to unit length, so every vector sits on the same sphere and only its direction varies. Many embedding models return normalised vectors by default: Titan Text Embeddings V2 takes an optional normalize parameter that defaults to true. Once every vector has length one, cosine similarity and dot product become the same computation, because the magnitude term is always one and drops out. Euclidean lines up too. On unit vectors, ranking by ascending Euclidean distance gives the same order as ranking by descending cosine similarity, so all three agree and the choice barely matters.

The danger is the other case. If the model emits vectors that are not normalised, and the semantic signal lives in the direction, then Euclidean distance and raw dot product let magnitude differences distort the ranking. A chunk that happens to produce a longer vector can crowd out a more relevant chunk with a shorter one, or the reverse, on length that carries no meaning. Nothing surfaces as an error. The query runs, results come back, and they are subtly wrong. RecallRecall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. drops, and the only symptom is that the answers are worse than they should be.

The last thing that matters is where the choice lives. It is not set on the model; it is set on the vector store, at index-creation time, and OpenSearch lists space_type as not updatable afterwards. In OpenSearch k-NN it is space_type, either at the top level of the knn_vector field or inside the method object. In pgvector it is which operator you query with and which operator class the index was built for. Get it right when you create the index, because changing it later means rebuilding.

What we’ll filter on

1. Does the embedding model normalise its output, or return raw-magnitude vectors?
2. Does magnitude carry meaning for this model, or is the signal in the direction alone?
3. Which metric does the model’s own documentation point to?
4. Which metrics does the target vector store expose, and what is its default?
5. Is the index setting a match for the model, or a default nobody chose?
6. Is the same embedding path used for both indexing and querying, so the vectors are comparable?

The landscape

Cosine similarity. Measures the cosine of the angle between two vectors, ranging from -1 (opposite) through 0 (orthogonal) to 1 (identical direction). It ignores magnitude and compares only orientation. OpenSearch names it cosinesimil and describes it as suiting text embeddings, where direction matters more than magnitude. When in doubt on a text model, cosine is the safe first pick.

Dot product (inner product). Multiplies the vectors component-wise and sums, folding both angle and magnitude into the score: same direction, longer vector, higher score. On raw vectors this ranks differently from cosine. On normalised vectors it ranks the same, with no length division to compute, which is why the OpenSearch documentation steers you to innerproduct over cosinesimil for vectors that already arrive at unit length.

Euclidean (L2) distance. The gap between the two points, so smaller is closer. OpenSearch’s l2 computes the squared distance, which orders results identically to the straight-line one. It is sensitive to magnitude, and unlike the two above it is a distance rather than a similarity, so the sort direction is reversed. L2 suits embeddings where absolute position and magnitude carry information. Put un-normalised, direction-only vectors under an L2 index and they rank partly on length that means nothing.

The stores expose the choice. In OpenSearch k-NN, space_type accepts l1, l2, linf, cosinesimil, innerproduct, hamming and hammingbit, and defaults to l2. The default engine is faiss, whose HNSW method has supported cosinesimil only since OpenSearch 2.19, alongside l2 and innerproduct. Under faiss, cosinesimil normalises vectors to unit length during indexing, so the stored values differ from the ones you sent. In pgvector the operator carries the choice, <=> for cosine distance, <#> for negative inner product, <-> for L2, with a matching operator class (vector_cosine_ops, vector_ip_ops, vector_l2_ops) on the index. Bedrock Knowledge Bases sit on top of these stores, and the AWS setup guidance differs by store: l2 for float embeddings in OpenSearch, vector_cosine_ops for Aurora pgvector, cosine or Euclidean for S3 Vectors. That range works because Titan and Cohere return unit-length vectors by default, where all three metrics rank alike.

Evaluation

Side by side

Reading the table against this scenario is direct. The model returns raw-magnitude vectors whose meaning is in the direction, so the l2 default the index inherited ranks partly on length. Cosine is the metric that discards that length, and on faiss it normalises the stored vectors as it indexes them.

The solution

The fix here is to stop ranking on length, and two routes get there. Rebuild the index with space_type set to cosinesimil, which discards magnitude and, on the faiss engine, normalises the stored vectors during indexing. Or set the model’s normalize parameter back to true, re-embed the corpus, and index under innerproduct, which on unit vectors ranks the same as cosine with less arithmetic. What must not stay in place is raw-magnitude, direction-only vectors under an L2 index, because that is where length reorders the results and recall drops. If you normalise, normalise on both sides, indexing and querying, or the two are not comparable.

Neither route is a small edit to a running system. OpenSearch lists space_type as not updatable after index creation, so changing the metric means creating a new index and reindexing into it, and changing normalize means calling the embedding model again over every chunk. That is the argument for setting the metric deliberately when the index is created rather than leaving it at the default.

This is worth care rather than a shrug because of the failure signature. A wrong metric does not throw, does not slow the query, and does not show up in any health check; it returns worse neighbours. The way you catch it is not monitoring but evaluation: a small labelled set of queries with known-relevant chunks, run through the pipeline, measuring whether the right chunks land in the top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost.. A metric mismatch shows up immediately as poor recall on that set, where it is invisible everywhere else. Build the eval set before you trust the retrieval, because it is the only instrument that sees this class of bug.

One more consistency trap sits underneath all of it: the same embedding model and the same normalisation must be used for both the indexed documents and the query. Embed the corpus with one model and the queries with another, or normalise one side and not the other, and the vectors live in incompatible spaces no matter how well the metric is chosen. The metric matches the model, and both sides of the search must run the same model.

Worked example

The index was created without an explicit metric, so OpenSearch applied its defaults: l2 for the space type, faiss for the engine. The mapping looked, in effect, like this:

PUT /chunks
{
  "settings": { "index": { "knn": true } },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "faiss"
        }
      }
    }
  }
}

Meanwhile the indexing job called Titan with normalisation switched off, so the vectors arrived at their raw lengths:

{
  "inputText": "...",
  "dimensions": 1024,
  "normalize": false
}

Under l2 the search still runs and still returns ten neighbours, so nothing looks wrong, but the ranking is computed on a mix of direction and length when only direction carries meaning. The corrected mapping names the metric, which can sit at the top level of the field:

PUT /chunks
{
  "settings": { "index": { "knn": true } },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "space_type": "cosinesimil",
        "method": {
          "name": "hnsw",
          "engine": "faiss"
        }
      }
    }
  }
}

That combination needs OpenSearch 2.19 or later, since faiss HNSW gained cosinesimil in that release; on an older cluster the equivalent is to normalise the vectors before indexing and use innerproduct. The same choice in pgvector is made not on the column but on the query operator and the index behind it: build the HNSWHNSWA graph-based vector index that walks neighbour links to find close vectors fast, at the cost of extra memory per vector. index with vector_cosine_ops and query with the <=> cosine-distance operator, so the approximate index and the search agree. In either store the documents did not change and the model did not change; only the store’s definition of “near” was brought back into line with the vectors it holds.

What’s worth remembering

1. The distance metric is a property of the embedding model, not a tuning knob; the index has to match what the model emits.
2. Cosine discards magnitude and suits direction-only text embeddings, while Euclidean and raw dot product rank partly on vector length.
3. On unit-length vectors, cosine and inner product are the same computation and Euclidean ranks identically, so the choice barely matters.
4. The dangerous case is un-normalised direction-only vectors under an L2 index, where length reorders results with no error, slowdown, or alert.
5. OpenSearch defaults space_type to l2 and the engine to faiss, faiss HNSW has supported cosinesimil only since 2.19, and neither setting can be changed without a reindex.
6. A wrong metric is a silent failure; the instrument that catches it is a small labelled evaluation set measuring whether relevant chunks land in the top-k.