The situation
A retail team has a catalogue of about 400,000 products, each with one or more photos and a short text description. They want three things from the same search box. A shopper should be able to type “red canvas high-top trainers” and get the right products back even when nobody wrote the words “high-top” into the description. A merchandiser should be able to upload a supplier photo and find visually similar items already in the range, to catch near-duplicates before they list them. And a “more like this” widget on the product page should surface visually related items regardless of how their descriptions were worded.
The first instinct on the team is to reach for the vision-capable chat model they already use on Amazon Bedrock, the one that can look at an image and describe it. Its descriptions are accurate. But wiring it into search means asking it, for every query, to compare against 400,000 products one at a time, which is neither affordable nor fast. Something is wrong with the shape of the tool, not the quality of it.
The job underneath all three features is the same: find the nearest items in a library, where the query might be text, might be an image, and the library is a mix of both. That is a retrieval problem, and retrieval runs on vectors.
What actually matters
The first thing to settle is whether the task is retrieval or reasoning, because the two need completely different tools. Retrieval means “of everything I have stored, which items are most like this one”, answered by turning both the query and the corpus into vectors and finding the nearest neighbours by distance. Reasoning means “look at this specific image and tell me something about it”, answered by a foundation model that takes the image into its context and generates a response. A multimodal chat model takes one image per call and returns text about it. It returns no vector to store, so it cannot build a searchable index, and it does not scale to comparing a query against hundreds of thousands of items. Embeddings build the index. The chat model handles one input at a time. Reaching for the chat model to do search is the mistake that makes everything slow and expensive.
Once it is a retrieval problem, the second thing that matters is the shared vector space. A text-only embedding model maps text to vectors, and two pieces of text that mean similar things land close together. A multimodal embedding model, such as Amazon Titan Multimodal Embeddings, maps both images and text into the same space, so a photo of red high-top trainers and the phrase “red high-top trainers” land near each other even though one is pixels and the other is words. That single shared space is what makes cross-modal search work: you embed the corpus of images once, and at query time you embed whatever the shopper gave you, text or image or both, and search the same index. A text-only model cannot do this, because it has no way to place an image anywhere in its space.
The third thing is that the vectors, whatever produced them, live in an ordinary vector store and are queried by an ordinary Nearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed.. There is nothing special about image vectors once they exist; they are floating-point arrays of a fixed length, and they go into the same k-nearest-neighbour index you would use for text-based retrieval. The image-search feature and the text-search feature share one store and one query path; only the input handed to the embedding model changes.
The fourth thing goes wrong without raising an error: the index has to be internally consistent. Every vector in it must come from the same embedding model at the same output dimension. AWS documents Titan Multimodal Embeddings vectors as comparable by Cosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”. or Euclidean distance, so either measure is defensible, but the index is built with one of them and every query uses the same one. Mix in vectors from a different model, or from the same model at a different Embedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching., and the distances mean nothing. The same goes for searching cosine vectors with a raw dot product on unnormalised data.
What we’ll filter on
- Retrieval or reasoning, are we finding nearest items in a library, or interpreting a single image in a prompt?
- Query and corpus modalities, is the query text, image, or both, and is the corpus text, image, or both?
- Shared space, does the search need image and text to sit in one comparable vector space, or is one modality enough?
- Metric and dimension match, does every vector come from the same model, at the same output dimension, searched with the one metric the index was created for?
- Store and scale, can the vector store hold the corpus and answer nearest-neighbour queries at the catalogue size and latency required?
The landscape
Text embedding model. A model such as Amazon Titan Text Embeddings turns text into a vector, and similar text lands nearby. It is the right tool for text-to-text semantic search, the retrieval half of a document-grounded assistant, and clustering or classification over text. It has no notion of images at all, so it cannot answer an image query or index a photo. If both the query and the corpus are text, this is the cheaper, simpler choice; the moment an image enters either side, it cannot help.
Multimodal embedding model. Amazon Titan Multimodal Embeddings G1, model ID amazon.titan-embed-image-v1, maps images and text into one shared space. It accepts a text input, an image input, or both together, and returns a vector in the same space every time. Text input is capped at 256 tokens; images at 25 MB and 2048 by 2048 pixels. Output length is 1024 by default, with 384 and 256 available for a size-versus-accuracy trade. Send text and an image in one call and the returned vector is the average of the text vector and the image vector, which is how a query like “this dress, but in blue” works. All three catalogue features need image and text comparable in one space, so all three sit on a model of this kind.
The newer Amazon embedding model. Amazon Nova Multimodal Embeddings, model ID amazon.nova-2-multimodal-embeddings-v1:0, launched in October 2025 and covers text, document images, still images, video and audio in one semantic space. Its context length is 8K tokens, or 30 seconds of video or audio. Output dimensions are 3072, 1024, 384 and 256. Small inputs go through InvokeModel synchronously; large files go through StartAsyncInvoke, which writes the embeddings to Amazon S3. It also takes an embedding purpose, so the vectors can be tuned for retrieval rather than classification or clustering. Its regional availability is much narrower than Titan’s, so read the model card before committing a catalogue to it. Cohere Embed v4 on Bedrock is a third option with the same shape of capability. The deciding constraint does not change: pick one model and populate the whole index with it.
Multimodal foundation model. A vision-capable chat model, such as the Claude and Amazon Nova chat models on Bedrock, takes an image in the prompt and reasons about it: describing it, answering questions about it, extracting fields, comparing it to something also in the prompt. This is understanding and generation, not retrieval. It is superb at “what is in this photo” and useless as a search index, because it processes one input per call and produces language, not a vector you can store and compare at scale. It has a real place around the edges of a search system (generating captions to enrich the corpus, or re-ranking a short candidate list the vector search already narrowed down), but it is not the thing that finds candidates in the first place.
The vector store and its metric. OpenSearch Service and OpenSearch Serverless, Aurora PostgreSQL and RDS for PostgreSQL with pgvector, Amazon MemoryDB, and Amazon S3 Vectors all hold vectors and answer k-nearest-neighbour queries. Amazon Bedrock Knowledge Bases can manage the ingestion-and-index path over a subset: OpenSearch Serverless and managed clusters, S3 Vectors, Aurora PostgreSQL, Neptune Analytics, and a few third-party stores. Take that route and Titan Multimodal is supported at 1024 dimensions only, so the smaller output lengths are off the table. The store choice is otherwise largely orthogonal to the modality question, because image vectors and text vectors are the same kind of object once produced. The metric is not orthogonal. It is fixed when the index is created, so a change of mind later means rebuilding the index.
Evaluation
Side by side
| Property | Text embedding | Multimodal embedding | Multimodal FM (chat) |
|---|---|---|---|
| Builds a searchable index | ✓ | ✓ | ✗ |
| Handles image queries | ✗ | ✓ | ✓ (reasons, not retrieves) |
| Handles text queries | ✓ | ✓ | ✓ (reasons, not retrieves) |
| Image and text in one shared space | ✗ | ✓ | n/a |
| Scales to nearest-neighbour over a large corpus | ✓ | ✓ | ✗ |
| Reasons about a single image | ✗ | ✗ | ✓ |
| Output | Vector | Vector | Text |
| Right job here | Text-only search | Cross-modal catalogue search | Captioning / re-ranking a shortlist |
The solution
For the three catalogue features, the pick is one multimodal embedding model over one shared index. Titan Multimodal Embeddings is available in more Regions; Nova Multimodal Embeddings covers more modalities and a larger maximum dimension where it runs. Embed every product image with the model you chose and store the vectors, with the product id and useful metadata, in a k-nearest-neighbour index. The three features then differ only in what goes into the model. Text search embeds the shopper’s phrase with the same model and searches; because the model shares a space across modalities, “red canvas high-top trainers” lands near the trainer photos even when the description never used those words. Reverse image search embeds the uploaded supplier photo and searches the same index for the nearest product images, which surfaces the near-duplicates. The “more like this” widget takes the current product’s own image vector, which is already in the index, and pulls its neighbours. One model, one store, three query paths.
The blended query is where the multimodal model helps most and where a text-only approach cannot reach. “This dress, but in blue” is an image plus a text refinement. Titan Multimodal takes both in one call and returns the average of the image vector and the text vector, so the result sits between the two. That combined-input capability is a property of the model, not something you can bolt on with a text embedder and a photo tagger.
The distance metric separates a working index from a subtly broken one. Pick one measure, cosine similarity or Euclidean distance, configure the store for it when the index is created, and make sure every vector came from one model at one output dimension. Knowledge Bases recommends Euclidean for floating-point embeddings when you hand-build the OpenSearch index; S3 Vectors offers cosine or Euclidean. Either serves, as long as nothing else in the index disagrees. Re-embed the catalogue at a different dimension, or add a second model for part of the corpus, and the old and new vectors stop being comparable. Nothing errors. The results simply get worse. A re-embed is an all-or-nothing migration of the whole index, not a per-item upgrade, and the smaller output dimensions exist for the size-versus-Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. trade at scale.
The multimodal chat model still has a role, just not the retrieval one. It is the right tool to generate a rich caption for each product at ingestion time, which enriches the metadata and can improve text search, and it is a sound choice to re-rank the top handful of candidates the vector search returned, where reasoning over a short list is affordable. What it must not be is the thing that scans the catalogue, because reasoning over every item per query is the cost and latency wall the team hit at the start.
Worked example
Ingestion runs once. Each of the 400,000 products has its image (or images) sent to Titan Multimodal Embeddings, and the returned 1024-dimension vector is written to an OpenSearch k-NNThe retrieval question itself: given a query vector, return the k closest vectors under the index’s distance metric – answered exactly by comparing against everything, or quickly by an ANN index. index configured for cosine similarity, alongside the product id, title, price, and category as metadata.
Query one, text. A shopper types “red canvas high-top trainers”. The phrase goes to the same model, comes back as a vector in the same space, and a cosine k-NN search returns the nearest product vectors, trainers whose photos sit near the phrase, even for a listing whose description only said “casual lace-up shoe”. The words the merchandiser never wrote do not matter, because the match happened in the shared space, not on keywords.
Query two, image. A merchandiser uploads a supplier photo of a jacket. It is embedded by the same model and searched against the same index, returning the visually nearest products; two of them are the same jacket already listed under different titles, which is the duplicate the merchandiser was hunting for. No text was involved on either side, and yet the query used the identical path.
Query three, blended. On a product page, the shopper clicks “in blue” under a dress. The dress image and the word “blue” go to the model in one call, and the averaged vector searches the same index, returning dresses shaped like the original but shifted toward blue. The result set is neither a pure image match nor a pure text match. One shared multimodal space produces that; a stack of single-modality tools does not.
Nowhere in the three did a chat model read the catalogue. It captioned products during ingestion and could re-rank the top ten results, but the search itself was cosine nearest-neighbour over vectors from one model in one index.
What’s worth remembering
- Search is retrieval, not reasoning; retrieval runs on embeddings and nearest-neighbour distance, and a chat model that reads one image per call cannot be the search index.
- A multimodal embedding model, such as Amazon Titan Multimodal Embeddings or the newer Amazon Nova Multimodal Embeddings, places images and text in one shared vector space, so a photo and a matching phrase land near each other.
- Every vector in an index must come from the same model at the same output dimension, and one metric, cosine or Euclidean, is fixed when the index is created and used by every query.
- Re-embedding at a different dimension or adding a second model breaks comparability; a re-embed is an all-or-nothing migration of the whole index.
- A multimodal foundation model still helps around the edges, captioning the corpus at ingestion and re-ranking a short candidate list, but never scanning the whole catalogue per query.