The situation
A retail team has a catalogue of about 400,000 products, each with one or more photos and a short text description. They want three things from the same search box. A shopper should be able to type “red canvas high-top trainers” and get the right products back even when nobody wrote the words “high-top” into the description. A merchandiser should be able to upload a supplier photo and find visually similar items already in the range, to catch near-duplicates before they list them. And a “more like this” widget on the product page should surface visually related items regardless of how their descriptions were worded.
The first instinct on the team is to reach for the vision-capable chat model they already use on Amazon Bedrock, the one that can look at an image and describe it. It reads a photo beautifully. But wiring it into search means asking it, for every query, to compare against 400,000 products one at a time, which is neither affordable nor fast. Something is wrong with the shape of the tool, not the quality of it.
The job underneath all three features is the same: find the nearest items in a library, where the query might be text, might be an image, and the library is a mix of both. That is a retrieval problem, and retrieval runs on vectors.
What actually matters
The first thing to settle is whether the task is retrieval or reasoning, because the two want completely different tools. Retrieval means “of everything I have stored, which items are most like this one”, answered by turning both the query and the corpus into vectors and finding the nearest neighbours by distance. Reasoning means “look at this specific image and tell me something about it”, answered by a foundation model that takes the image into its context and generates a response. A multimodal chat model reads one image per call and thinks about it; it does not build a searchable index, and it does not scale to comparing a query against hundreds of thousands of items. Embeddings build the index; the chat model interprets a single input. Reaching for the chat model to do search is the mistake that makes everything slow and expensive.
Once it is a retrieval problem, the second thing that matters is the shared vector space. A text-only embedding model maps text to vectors, and two pieces of text that mean similar things land close together. A multimodal embedding model, such as Amazon Titan Multimodal Embeddings, maps both images and text into the same space, so a photo of red high-top trainers and the phrase “red high-top trainers” land near each other even though one is pixels and the other is words. That single shared space is what makes cross-modal search work: you embed the corpus of images once, and at query time you embed whatever the shopper gave you, text or image or both, and search the same index. A text-only model cannot do this, because it has no way to place an image anywhere in its space.
The third thing is that the vectors, whatever produced them, live in an ordinary vector store and are queried by an ordinary Nearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed. . There is nothing special about image vectors once they exist; they are floating-point arrays of a fixed length, and they go into the same k-nearest-neighbour index you would use for text-based retrieval. This is the part that surprises people: the image-search feature and the text-search feature share one store and one query path, differing only in which model produced the query vector.
The fourth thing, and the one that quietly breaks systems, is that the model and the distance metric have to match. Every vector in the index must come from the same embedding model at the same output dimension, and the search must use the distance measure that model was built for. Titan Multimodal Embeddings is built for Cosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”. ; mixing in vectors from a different model, or from the same model at a different Embedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. , or searching cosine vectors with a raw dot product on unnormalised data, produces distances that are meaningless. The index is only coherent if one model populated it and one metric reads it.
What we’ll filter on
- Retrieval or reasoning, are we finding nearest items in a library, or interpreting a single image in a prompt?
- Query and corpus modalities, is the query text, image, or both, and is the corpus text, image, or both?
- Shared space, does the search need image and text to sit in one comparable vector space, or is one modality enough?
- Metric and dimension match, does every vector come from the same model, at the same output dimension, searched with the metric that model expects?
- Store and scale, can the vector store hold the corpus and answer nearest-neighbour queries at the catalogue size and latency required?
The embedding landscape
Text embedding model. A model such as Amazon Titan Text Embeddings turns text into a vector, and similar text lands nearby. It is the right tool for text-to-text semantic search, the retrieval half of a document-grounded assistant, and clustering or classification over text. It has no notion of images at all, so it cannot answer an image query or index a photo. If both the query and the corpus are text, this is the cheaper, simpler choice; the moment an image enters either side, it cannot help.
Multimodal embedding model. Amazon Titan Multimodal Embeddings maps images and text into one shared space. It accepts a text input, an image input, or an image and text together, and returns a vector in the same space every time. That means you can search images with a text query, search with an example image, or combine an image with a text refinement (“this dress, but in blue”) and search on the blended vector. Output dimensionality is configurable, commonly 1024 with smaller options like 384 and 256 for a size-versus-accuracy trade, and the model is built to be compared with cosine similarity. This is the tool for every one of the three catalogue features, because all three need image and text to be comparable in one space. Cohere Embed on Bedrock also offers image-and-text embeddings as an alternative model family with the same shape of capability; the deciding discipline is the same, pick one and populate the whole index with it.
Multimodal foundation model. A vision-capable chat model, such as the Claude or Amazon Nova models on Bedrock, takes an image in the prompt and reasons about it: describing it, answering questions about it, extracting fields, comparing it to something also in the prompt. This is understanding and generation, not retrieval. It is superb at “what is in this photo” and useless as a search index, because it processes one input per call and produces language, not a vector you can store and compare at scale. It has a real place around the edges of a search system (generating captions to enrich the corpus, or re-ranking a short candidate list the vector search already narrowed down), but it is not the thing that finds candidates in the first place.
The vector store and its metric. OpenSearch Service and OpenSearch Serverless, Aurora PostgreSQL and RDS for PostgreSQL with pgvector, Amazon MemoryDB, and the newer purpose-built vector options all hold vectors and answer k-nearest-neighbour queries; Amazon Bedrock Knowledge Bases can manage the ingestion-and-index path over several of them. The store choice is largely orthogonal to the modality question, because image vectors and text vectors are the same kind of object once produced. What is not orthogonal is the metric: the store has to be configured for the distance the embedding model expects, cosine similarity for Titan Multimodal, and every vector in it has to come from that one model at one dimension.
Side by side
| Property | Text embedding | Multimodal embedding | Multimodal FM (chat) |
|---|---|---|---|
| Builds a searchable index | ✓ | ✓ | ✗ |
| Handles image queries | ✗ | ✓ | ✓ (reasons, not retrieves) |
| Handles text queries | ✓ | ✓ | ✓ (reasons, not retrieves) |
| Image and text in one shared space | ✗ | ✓ | n/a |
| Scales to nearest-neighbour over a large corpus | ✓ | ✓ | ✗ |
| Reasons about a single image | ✗ | ✗ | ✓ |
| Output | Vector | Vector | Text |
| Right job here | Text-only search | Cross-modal catalogue search | Captioning / re-ranking a shortlist |
The picks in depth
For the three catalogue features, the pick is one multimodal embedding model over one shared index. Embed every product image with Titan Multimodal Embeddings and store the vectors, with the product id and useful metadata, in a k-nearest-neighbour index. The three features then differ only in what produces the query vector. Text search embeds the shopper’s phrase with the same model and searches; because the model shares a space across modalities, “red canvas high-top trainers” lands near the trainer photos even when the description never used those words. Reverse image search embeds the uploaded supplier photo and searches the same index for the nearest product images, which surfaces the near-duplicates. The “more like this” widget takes the current product’s own image vector, which is already in the index, and pulls its neighbours. One model, one store, three query paths.
The blended query is where the multimodal model earns its keep and where a text-only approach could never reach. “This dress, but in blue” is an image plus a text refinement; Titan Multimodal can embed the image and the phrase together into a single vector that leans toward the visual match while nudging on the colour. That combined-input capability is a property of the model, not something you can bolt on with a text embedder and a photo tagger.
The distance metric is the detail that separates a working index from a subtly broken one. Configure the store for cosine similarity, the measure Titan Multimodal is built for, and make sure every vector in the index came from that model at one chosen output dimension. If the catalogue is later re-embedded at a different dimension, or a second model is introduced for part of the corpus, the old and new vectors are no longer comparable and the search quality quietly rots; a re-embed is an all-or-nothing migration of the whole index, not a per-item upgrade. The smaller output dimensions exist precisely for the size-versus-Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. trade at scale, but the choice is made once for the whole index.
The multimodal chat model still has a role, just not the retrieval one. It is the right tool to generate a rich caption for each product at ingestion time, which enriches the metadata and can improve text search, and it is a sound choice to re-rank the top handful of candidates the vector search returned, where reasoning over a short list is affordable. What it must not be is the thing that scans the catalogue, because reasoning over every item per query is the cost and latency wall the team hit at the start.
A worked example: three queries, one index
Ingestion runs once. Each of the 400,000 products has its image (or images) sent to Titan Multimodal Embeddings, and the returned 1024-dimension vector is written to an OpenSearch k-NN index configured for cosine similarity, alongside the product id, title, price, and category as metadata.
Query one, text. A shopper types “red canvas high-top trainers”. The phrase goes to the same model, comes back as a vector in the same space, and a cosine k-NN search returns the nearest product vectors, trainers whose photos sit near the phrase, even for a listing whose description only said “casual lace-up shoe”. The words the merchandiser never wrote do not matter, because the match happened in the shared space, not on keywords.
Query two, image. A merchandiser uploads a supplier photo of a jacket. It is embedded by the same model and searched against the same index, returning the visually nearest products; two of them are the same jacket already listed under different titles, which is the duplicate the merchandiser was hunting for. No text was involved on either side, and yet the query used the identical path.
Query three, blended. On a product page, the shopper clicks “in blue” under a dress. The dress image and the word “blue” are embedded together into one vector, and the search returns dresses shaped like the original but shifted toward blue in the space. The result set is neither a pure image match nor a pure text match, which is exactly what a single shared multimodal space allows and a stack of single-modality tools cannot.
Nowhere in the three did a chat model read the catalogue. It captioned products during ingestion and could re-rank the top ten results, but the search itself was cosine nearest-neighbour over vectors from one model in one index.
What’s worth remembering
- Search is retrieval, not reasoning; retrieval runs on embeddings and nearest-neighbour distance, and a chat model that reads one image per call cannot be the search index.
- A multimodal embedding model, such as Amazon Titan Multimodal Embeddings, places images and text in one shared vector space, so a photo and a matching phrase land near each other.
- That shared space is what makes cross-modal search work: text queries against image corpora, image queries against image corpora, and blended image-plus-text queries all use one index.
- A text-only embedding model cannot index or query images at all; use it when both the query and the corpus are text, and reach for a multimodal model the moment an image enters either side.
- Image vectors are ordinary vectors once produced; they live in the same k-nearest-neighbour store and query path as text vectors, differing only in which model made them.
- Every vector in an index must come from the same model at the same output dimension, and the search must use the metric that model expects, cosine similarity for Titan Multimodal.
- Re-embedding at a different dimension or adding a second model breaks comparability; a re-embed is an all-or-nothing migration of the whole index.
- A blended query, an image plus a text refinement embedded together, is a property of the multimodal model that no text-embedder-plus-tagger arrangement can reproduce.
- A multimodal foundation model still helps around the edges, captioning the corpus at ingestion and re-ranking a short candidate list, but never scanning the whole catalogue per query.
- Decide first whether you are retrieving or reasoning, then which modalities the query and corpus use; those two questions pick the model before any service does.