Why embeddings don't work for your search: 7 failure modes and how to fix them

When embeddings don't work for your search, it rarely looks like an outage. Results just feel slightly off. Users rephrase, add words, give up. Your RAG assistant answers confidently from the wrong document. And because nothing is technically broken, nobody can say why.

The good news: embedding failures are not random. After twelve years of building search at Walmart, Pinterest, Roblox and Shopify, I see the same seven patterns again and again, and each has a well-understood fix. This guide walks through how to diagnose which one you have, then how to fix it, in the order that gives you the most improvement for the least work.

In short: embeddings rank by overall similarity, so they miss exact identifiers, negation, numbers and multi-part constraints, and they never say "no results". Find where the right document drops out, then add keyword search, query understanding and a reranker before you touch the embedding model.

Why this keeps catching teams out

At Walmart, we were constantly answering the same question from sellers: why isn't my item showing up? Back then, most of the time the answer was a typo. A listing might say "bluetooh speaker" while shoppers typed "bluetooth speaker", so exact match failed and the item was never retrieved. That's how much we depended on good keywords.

Other times, a stray phrase threw off the ranking. Phrase matches (bigrams and longer n-grams) always got extra weight, whether or not those words mattered to the query, and working out which terms in a query really matter was a hard problem. So we maintained a tool where merchants and sellers could tag the right tokens and keywords for their items by hand. It became a superpower for controlling ranking and, frankly, for spamming it.

Keyword search failed in ways you could explain, and fix one listing at a time. Then embeddings arrived, promising to match meaning instead of strings.

In the late 2010s, embeddings became the hottest thing in search. Approximate nearest-neighbour algorithms like HNSW made it fast enough to search millions of vectors in production, libraries like FAISS made it easy, and every search and e-commerce team I knew wanted to replace keyword search with "semantic" search.

Slowly, and painfully, most of them learned that you can't simply swap one for the other. Two reasons came up again and again.

Users are trained to type keywords. Decades of search boxes have taught people to type a brand, a model number, two nouns. Embeddings shine on descriptive, sentence-like queries. On the queries people actually type, their results often feel loose and surprising.

Embedding search never says "no results". When it works, it's wonderful: it understands what you meant, not just what you typed. When it fails, it fails badly, and silently. Keyword search at least admits defeat: no exact match, "no results found". Vector search always returns the nearest neighbours, however far away they are. And you can't fix that with a similarity threshold. Search traffic follows a power law, and a score that means "good match" for a frequent query means something quite different for a rare one. Tune the cutoff for one and you break the other.

The quick diagnosis

Before changing anything, find out where search breaks. Take 50 to 100 real queries where users were unhappy (rephrased, clicked nothing, or flagged an answer), and for each one note the document that should have come back. Then, for each query, here's how I debug it today:

1. Was it retrieved at all? Turn off everything after retrieval: ML ranking, reranking, business boosts. Then run your normal text-match query with a filter on top that keeps only the expected document. If it comes back, retrieval matched it and the problem is ordering: a ranking problem, which a reranker usually fixes (see fix 4 below). If it doesn't, it was never a candidate. Your search engine's debugging tools earn their keep here: Elasticsearch and OpenSearch, for example, have an explain API that tells you why a document did or didn't match. One warning: if your vectors live in an HNSW index, filtered search can be painfully slow or quietly miss documents that do match, so use exact (brute-force) vector search for this check.
2. Which candidate generator should have found it? If the query has exact-match intent, like a SKU, a model number or a name, your keyword search should return it. If it doesn't, the fix is in your keyword indexing (tokenization, analyzers, which fields you index), not your embeddings. If the query is descriptive, look at the embedding side instead.
3. Can the embedding find it with exact search? If the right document isn't in the top 100 even with brute-force search, the embedding can't represent what the query needs. That's a representation problem: the seven failure modes below.
4. Does exact search find it, but your vector index doesn't? Then it's an index problem, not an embedding problem. Approximate nearest-neighbour indexes like HNSW trade recall for speed. Raise ef_search (or your index's equivalent) and measure recall against brute force before you blame the model.

This takes an afternoon and changes everything that follows. Teams that skip it often spend weeks fine-tuning an embedding model to fix what was actually an index setting.

Now, the seven ways representation breaks.

1. Exact identifiers: SKUs, part numbers, names

Symptom: a user searches WX-4410-B and gets WX-4410-C, WX-4401-B and a blog post about model numbers.

Embedding models split rare strings into sub-word pieces and map them to something like the right region of the space. "Close" is fine for meaning, but useless for identifiers, where one character is the difference between the right part and the wrong one. Product codes, error codes, ticket numbers, person and company names, and version strings all fail this way.

The research agrees. In Simple Entity-Centric Questions Challenge Dense Retrievers (Sciavolino et al., EMNLP 2021), a widely used dense retriever (DPR) found the right passage for simple questions about named entities 49.7% of the time, against 72.0% for plain BM25. On questions like "Where was [person] born?" the gap was almost 50 points. Dense models handled entities they'd seen often in training, and struggled with the long tail. Your product catalog is mostly long tail.

Fix: keyword search. BM25 matches exact tokens and gives rare tokens a high weight, which is exactly what identifiers need. Many teams also add a fast path: if the query looks like an identifier (a regex is usually enough), do an exact lookup first and only fall back to semantic search when it finds nothing.

Better still, give identifiers their own fields. Instead of leaving SKUs, model numbers and brand names buried in titles and descriptions, index each in a dedicated field (sku, model_number, brand) with exact matching, not the usual word-splitting, and give a match on those fields priority. As a bonus, it helps your ranking model: "the query matched the model number field" is one of the strongest signals a learning-to-rank model can get.

2. Negation: "boots without laces"

Symptom: "boots without laces" returns lace-up boots. "Flights not via Frankfurt" returns flights via Frankfurt. "Dairy-free dessert" ranks dairy desserts highly.

An embedding compresses a whole query into one point. "Boots without laces" and "boots with laces" share almost every word and the same topic, so they land very close together. The model registers laces far more strongly than without. This is well documented: NevIR (Weller et al., EACL 2024) tested this directly: each test is a pair of near-identical documents where one negation flips the meaning, and the model has to rank the right one higher for each query. Most neural retrieval models, including state-of-the-art ones, did no better than random. Standard embedding models (bi-encoders) did worse than random.

Fix: treat negation as structure, not meaning. Detect exclusions in the query ("without", "no", "-free", "except", "not") and turn them into filters on your catalog or document attributes. This is the fix that actually works. Rerankers help, but less than you'd hope: in NevIR, most cross-encoders scored only around random, and even the best, a 3-billion-parameter model, got half the pairs right. Negation is a problem to solve in query understanding, not to hope the model learns.

3. Numbers and units: "under $50", "size 11"

Symptom: "running shoes under $50" returns $150 shoes. "Size 11" returns size 1. "Laptops with 32GB RAM" returns 16GB models whose description mentions 32GB as an upgrade option.

Embedding models see numbers as tokens, not quantities. Probing studies (Wallace et al., EMNLP 2019) found that some sense of magnitude does survive in embeddings, but sub-word models like BERT encode it imprecisely, and it breaks down outside the ranges seen in training. More importantly, even a perfect sense of magnitude wouldn't help: a similarity score has no concept of "under". "Under $50" and "over $50" are nearly the same vector, and $50 reads as a topic, not a price ceiling. Units make it worse: "11" means different things in "size 11", "iOS 11" and "11 inch".

Fix: extract numbers into structured filters. Parse prices, sizes, dates and quantities from the query (rules handle the common cases; a small LLM call handles the long tail), and apply them as hard filters or range constraints in your index. Then let the embedding do what it's good at: the meaning of the rest of the query.

This is harder than it looks, because it depends entirely on query understanding. The parser has to know what each number is. In "iphone 15 case", 15 is a model, not a quantity. In "2 pack socks" it's a quantity, and in "under 50" it's probably a price. Get it wrong and a hard filter removes every right answer, which is worse than no filter at all. So use the attributes you're confident about as filters, and the rest as gentler ranking boosts.

For years, query understanding was an afterthought in many search stacks. Now that AI agents are searching on people's behalf, sending long queries packed with prices, sizes and dates, it has become one of the most important parts of the system.

4. Hard constraints: "waterproof trail running shoes, wide fit"

Symptom: every result is mostly right. They're trail running shoes, but not waterproof. Or waterproof, but not wide fit. Users scroll, find nothing that meets every requirement, and leave.

This is the same compression problem in a subtler form. A query with several requirements becomes one vector that sits somewhere in the middle of all of them. Documents that strongly match two of the three requirements can score higher than the one document that matches all three. Embeddings rank by overall similarity; your user needs every constraint met.

Fix: the classic approach is query understanding. Break the query into its attributes (category: trail running shoes; feature: waterproof; fit: wide), map each to the structured data you already have, and use the attributes you're confident about as filters and the rest as boosts. It works, but it's a lot to build and maintain.

In practice, multiple candidate generators handle much of this on their own. Keyword search finds the products that literally say "waterproof" and "wide"; embeddings find the trail running shoes. A ranking model then combines them, and learns from engagement (clicks, add-to-carts, orders) which products actually satisfy queries like this one, so the shoe that meets every requirement rises to the top.

If you're starting cold, with no engagement data yet, give descriptors their own field. Index attributes like "waterproof", "wide fit" and "Gore-Tex" in a dedicated field, match query words against it, and feed those matches to your ranking model as features. Now the ranker can tell "matches all three" from "matches two", which a single similarity score can't.

This is one of the clearest examples of why embedding search alone isn't enough for e-commerce.

5. Short queries vs long documents

Symptom: two-word queries return documents that merely mention the topic, while the definitive page on that topic ranks lower. Or queries phrased as questions match other questions (FAQ titles, forum posts) instead of the answers.

Queries and documents are different kinds of text. A query is three words; a document is three paragraphs. A question looks more like another question than like its answer. Symmetric embedding models, trained to put similar texts close together, don't know that your query and your ideal document are supposed to look different.

Fix: first, check that you're using your model the way it was trained. Many retrieval models expect a prefix or instruction (for example "query: " and "passage: " for the E5 family, or a task instruction for instruction-tuned models). Leaving it off is one of the most common bugs I see, and it quietly costs a lot of quality. Second, consider enriching what you index: add titles, headings and generated example questions to each document, so short queries have something closer to match against.

6. Domain language the model never saw

Symptom: your internal acronyms, product names and jargon retrieve nonsense. In a medical, legal or industrial corpus, results for specialist terms look like the model is guessing.

General-purpose embedding models learned language from the public web. If your users say "PDP" and mean product detail page, while the model mostly saw it meaning something else, it will place it in the wrong part of the space. Brand-new product names, internal team names and codes simply have no learned meaning at all.

This is the core finding of BEIR (Thakur et al., NeurIPS 2021), the standard benchmark for retrieval on unfamiliar domains. Across 18 datasets, from biomedical papers to financial questions and scientific claims, dense retrievers that did well on the data they were trained on often fell behind BM25 on domains they hadn't seen. Newer embedding models have closed much of that gap on BEIR itself, but the lesson holds for your domain: a public leaderboard score tells you little about your jargon.

Fix: in order of effort: add a synonym and acronym expansion step (cheap, and surprisingly effective); use hybrid search so exact jargon still matches through BM25; and only then consider fine-tuning the embedding model on your own data (fix 5 below).

7. Chunks that average away the answer

Symptom (mostly in RAG): the answer is definitely in your documents, but retrieval returns the wrong chunk, or a chunk that's about the right topic without containing the actual answer.

Each chunk becomes one vector, which is roughly an average of everything in it. Literally so for many models: Sentence-BERT and many of its successors produce the chunk vector by mean pooling, averaging the vectors of every token. Models that use a single summary token instead still have to squeeze the whole chunk into one point. A 1,000-token chunk covering five topics produces a vector that is a bit like all five and strongly like none of them. The one sentence that answers the user's question gets diluted by everything around it. Dense retrievers also have measurable biases here: a 2025 study found popular retrievers prefer shorter documents, information near the start of a passage, and literal word overlap, often over the passage that actually contains the answer.

Chunks that are too small fail the other way: they lose the context that made them relevant ("it supports up to 32 devices" — what does?).

Fix: chunk along the structure of your documents (sections, headings, list items) instead of by fixed token counts. Prepend each chunk with its document title and section heading, so it carries its context with it. Then measure: try two or three chunk sizes on your evaluation queries and keep the one that wins. There is no universal right size.

This isn't only a retrieval trick. It's also how I prepared documents at Shopify for continued pretraining, or domain-adaptive pretraining (DAPT). A chunk that follows the document's structure and carries its own title and heading is just as useful to a model learning your domain as it is to a retriever.

The theoretical ceiling

Some of these failures aren't just engineering gaps; they're mathematical limits. A 2025 paper from Google DeepMind and Johns Hopkins, On the Theoretical Limitations of Embedding-Based Retrieval (ICLR 2026), shows that the number of distinct top-k result sets a single-vector embedding can represent is bounded by its dimension. Put simply, a single vector has a fixed capacity: some combinations of "these documents, and not those" cannot be expressed by any embedding of a given size, no matter how well it's trained. The more distinct concepts a query has to intersect, the sooner you hit that bottleneck.

The authors built a dataset (LIMIT) where the task is deliberately simple. On it, state-of-the-art single-vector embedding models, including 7B to 8B parameter ones, failed to reach even 20% recall@100. Plain BM25, which works in a very high-dimensional sparse space, scored 93.6%. A multi-vector model (GTE-ModernColBERT), which keeps one vector per token instead of one per document, landed in between at 54.8%.

What this means in practice: a better embedding model will not make every problem go away. Queries that combine several constraints (failure modes 2, 3 and 4) are exactly the kind that push against this ceiling. That's why the fixes that work best add other signals, such as keywords, structured filters and rerankers, instead of only swapping the model.

Keyword vs embeddings is the wrong argument

Search relevance has always been hard. Lucene-based engines like Solr and Elasticsearch spent years getting token and phrase matching right, first with TF-IDF and then with BM25, which only became Lucene's default in 2016. Then vector search arrived, and the field split into two camps.

The keyword camp had been doing search for decades, and it showed. Many trained their embedding models on labels from strict keyword-matching rules. The models learned exactly what they were taught, and ended up behaving like keyword search: an expensive second copy of what they already had.

The embedding camp believed that with a big enough model, think OpenAI scale, keywords would stop mattering. The LIMIT results above show why that doesn't hold: some queries are beyond what any single vector can represent, however large the model.

Both camps are wrong. Keyword search and embeddings are two candidate generators, and a good search system often needs more than two: structured lookups on attributes like price and size, and behavioural signals like what people clicked for this query before. Each finds results the others miss, and a reranker puts them in order. There is no silver bullet. The right mix depends on your queries and your documents, and in my experience you only find it by measuring, through a fair number of failed experiments. Not from a hunch, and not by listening to a hardened Solr loyalist or an embeddings evangelist.

What the usual advice gets wrong

If you search for why embeddings don't work, you'll often read that the problem is the "curse of dimensionality" or "polysemy" (one word with several meanings). In production search, those are rarely the real cause.

• The curse of dimensionality is a statement about random data in high dimensions. Learned embeddings aren't random: the meaningful structure sits in a much smaller space, and nearest-neighbour search on them works well in practice. When retrieval fails, a smaller model rarely fixes it.
• Polysemy was a real problem for older static word vectors. Modern embedding models read the whole query in context and handle "apple the fruit" vs "Apple the company" well whenever the query gives any hint.

The real culprits are the seven above: exact strings, negation, numbers, multiple constraints, query–document mismatch, unfamiliar vocabulary and chunking. Each is about information that a single similarity score throws away.

Fixes, in the order to try them

When embeddings don't work, it's tempting to reach for the most sophisticated fix first. Do the opposite. These are ordered from most improvement per unit of work to least.

1. Measure first. Build the evaluation set from the diagnosis above and track recall@k and nDCG on it. Without this, you can't tell whether any change helped. Even this can fail if an LLM judge is your only source of relevance labels: you end up making your results likeable to the LLM, not useful to your users. Check the judge against human labels and real engagement (clicks, orders) before you trust it.
2. Add keyword search (hybrid search). Run BM25 alongside vector search and merge the results (how to combine them properly), for example with reciprocal rank fusion. This alone fixes most identifier and jargon failures, and it's usually a few days of work. The hard part is the merge: a blend tuned for a model-number query like QN65Q80C hurts a descriptive one like "summer dresses for a trip to greece", and the other way round. Let the type of query decide the weighting, or let a reranker order the merged candidates.
3. Add query understanding. Extract filters (price, size, dates, exclusions) and attributes from queries and apply them as structured constraints. This fixes negation, numbers and multi-constraint queries, which embeddings alone can't.
4. Add a reranker. Retrieve 50 to 100 candidates, then rescore them with a cross-encoder that reads query and document together. It's slower per document, but much more precise.
5. Fine-tune the embedding model, last. It helps with domain language and with how your users phrase things, but it needs a few thousand good training pairs, including hard negatives, and a solid evaluation set to prove it worked. Done before steps 1 to 4, it often improves the metric you watch while hurting the ones you don't.

Most teams I've worked with get the majority of their improvement from steps 1 to 3, and never need step 5.

And save your money. It's tempting to send every query and its candidates to ChatGPT or Claude and let a frontier model do the ranking. For most of the failures in this article, that means paying per query, forever, for something a filter, an identifier field or a small reranker fixes once. Use large models where they earn their cost, not as your ranking layer by default.

Not sure which of these your search needs? That's what a relevance audit is for: we find where your search breaks and the cheapest fix for each failure, before you spend on anything bigger.

FAQ

Why does vector search return irrelevant results?

Vector search returns the documents whose overall meaning is closest to the query, which is not the same as the documents that meet every requirement in it. It is weak at exact identifiers, negation, numbers and queries with several constraints. Check first whether your approximate index is missing results that exact search would find.

Are embeddings bad at exact keyword matching?

Yes. Embedding models map rare strings like product codes, error codes and names to approximate meanings, so near-misses score almost as well as exact matches. Keyword search such as BM25 handles exact matching well, which is why hybrid search combines the two.

Can embeddings replace keyword search?

Not on their own. Embeddings are better at meaning and paraphrase, but worse at exact identifiers, negation and numbers, and they never return "no results". Most production search systems run both and merge the results, often with a reranker on top.

Should I fine-tune my embedding model?

Usually not first. Hybrid search, query understanding and a reranker fix most retrieval problems with less effort and less risk. Fine-tune when you have a solid evaluation set, a few thousand good training pairs, and evidence that the remaining failures come from domain vocabulary.

Is vector search enough for RAG?

Rarely on its own. Production RAG systems usually combine vector search with keyword search, metadata filters and a reranker, and measure retrieval separately from the final answer. Most wrong RAG answers start as retrieval failures, not generation failures.