Semantic Search: Beyond Keyword Matching

Your users do not search with your vocabulary. They type "laptop won't turn on" while your knowledge base says "power cycle troubleshooting," and keyword search returns nothing. Semantic search closes that gap, but only if you understand what is actually happening under the hood, because a naive vector search often ships worse results than the BM25 index it replaced.

Where keyword search breaks down

Classic search engines (Elasticsearch, OpenSearch, Postgres full text) rank documents with lexical scoring, usually BM25. The mechanics are simple: a document scores high when it contains the query terms, weighted by how rare those terms are across the corpus and normalized by document length.

BM25 is fast, cheap, explainable, and shockingly hard to beat on exact matches. But it has two structural blind spots:

• Vocabulary mismatch. "Cheap flights" and "affordable airfare" share zero tokens. To BM25 they are unrelated. Synonym dictionaries patch this, but they rot, they miss phrasing, and nobody maintains them past the first quarter.
• No sense of intent. "Apple charger" and "apple pie charger plate" both match "apple" and "charger." BM25 cannot tell that one query is about electronics and the other is tableware, because it never models meaning, only token statistics.

Stemming, n-grams, and query expansion push the ceiling up a little. They do not fix the core problem: lexical search matches strings, not concepts.

What embeddings actually do

Semantic search replaces string matching with geometry. An embedding model maps text to a dense vector, typically 384 to 3072 floating point dimensions, such that texts with similar meaning land close together in that space. Search becomes: embed the query, then find the nearest document vectors by cosine similarity or dot product.

The models that matter in practice:

• Hosted: OpenAI text-embedding-3-small and text-embedding-3-large, Cohere embed-v3, Voyage. Strong quality, zero ops, per-token pricing.
• Open weight: the bge, e5, and gte families, plus multilingual variants. Run them on your own GPU or CPU, keep data in-house, pay only for compute.

Three practical details engineers routinely miss:

1. Embeddings are model-locked. Vectors from two different models (or two versions of the same model) live in incompatible spaces. Changing models means re-embedding the entire corpus. Budget for it.
2. Dimension count is a knob, not a virtue. Matryoshka-trained models let you truncate vectors (say 3072 down to 512) and trade a little recall for a large drop in index size and latency. In our experience, most product search workloads barely notice the difference at 512 to 768 dimensions.
3. Asymmetric prefixes matter. Models like e5 expect query: and passage: prefixes. Skip them and retrieval quality quietly degrades with no error to tell you why.

If you are building retrieval to feed an LLM rather than a search results page, the same machinery applies. We cover that pipeline in depth in RAG systems explained.

The retrieval stack, end to end

A production semantic search system is a pipeline, and every stage can ruin the output of the stages after it:

1. Ingest and chunk. Split documents into retrievable units.
2. Embed. Batch chunks through the embedding model.
3. Index. Store vectors in an ANN (approximate nearest neighbor) index.
4. Retrieve. Embed the query, pull top-k candidates, usually 50 to 200.
5. Rerank. Rescore candidates with a heavier model, return the top 5 to 20.
6. Present. Snippets, highlighting, filters, and feedback capture.

Exact nearest neighbor search over millions of vectors is too slow, so every serious system uses an ANN index. The dominant algorithm is HNSW (hierarchical navigable small world graphs). Its three parameters are worth understanding because they are the levers you will actually pull:

# Typical HNSW starting point (Qdrant, pgvector, Weaviate all expose these)
hnsw:
  m: 16                 # graph connectivity: higher = better recall, more RAM
  ef_construct: 200     # build-time effort: higher = better graph, slower indexing
  ef_search: 100        # query-time effort: higher = better recall, slower queries

HNSW trades memory for speed. IVF-based indexes trade recall for memory. Product quantization compresses vectors 4x to 32x at a measurable recall cost. Which combination fits depends on corpus size, update rate, and latency budget. Picking the storage engine itself (pgvector inside your existing Postgres versus a dedicated engine like Qdrant, Weaviate, or Milvus) is a separate decision with its own tradeoffs, and we wrote a full guide on it: choosing a vector database.

Hybrid search is not optional

Here is the uncomfortable truth teams discover in week three: pure vector search loses to BM25 on a large class of queries. Part numbers, SKUs, error codes, person names, exact phrases, legal citations. The embedding model smears "ERR_CONNECTION_REFUSED" into a fuzzy cloud of networking concepts, while BM25 nails the exact string instantly.

The fix is hybrid retrieval: run lexical and vector search in parallel, then fuse the ranked lists. Reciprocal rank fusion (RRF) is the standard because it needs no score normalization:

def rrf(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return dict(sorted(scores.items(), key=lambda x: -x[1]))

fused = rrf([bm25_results, vector_results])

RRF only cares about rank positions, so BM25 scores and cosine similarities never need to live on the same scale. Most vector databases and OpenSearch now ship RRF natively. If yours does not, the function above is the whole algorithm.

Our default recommendation: start hybrid from day one. The lexical leg costs almost nothing, and it is your safety net for every query type the embedding model handles badly.

Chunking: where most projects quietly fail

Embedding models have context limits, and long documents dilute meaning: a 40-page manual embedded as one vector is an average of everything and a match for nothing. So you chunk. How you chunk determines your ceiling.

• Fixed-size chunks (300 to 800 tokens, 10 to 15 percent overlap) are the baseline. Simple, predictable, decent.
• Structure-aware chunking splits on headings, list boundaries, and paragraphs instead of raw token counts. It keeps ideas intact and almost always beats fixed-size splitting on real corpora.
• Contextual enrichment prepends the document title, section path, and key metadata to each chunk before embedding. A chunk that begins "Refund policy > EU customers > Digital goods:" retrieves far better than the bare paragraph.
• Parent-child retrieval indexes small chunks for precision but returns the surrounding section for display or LLM context. Small to match, large to read.

Store metadata (source, date, language, product line, access level) alongside every vector and filter on it at query time. Pre-filtering inside the ANN index is how you make "search only in my workspace" both correct and fast, and it is also your access control boundary. Never rely on post-filtering for permissions.

Reranking: the cheapest quality win available

Bi-encoder retrieval (embed query and documents separately) is fast but coarse. A cross-encoder reranker reads the query and each candidate document together and outputs a relevance score. It is far more accurate and far too slow to run over the whole corpus, which is exactly why the two-stage design exists: retrieve 100 candidates cheaply, rerank the 100 precisely.

Options that work today: Cohere Rerank, Voyage rerank models, or open-weight bge-reranker variants self-hosted on a modest GPU. Typical added latency is tens of milliseconds for a batch of 100 candidates, and in our experience it is the single highest-leverage upgrade after hybrid retrieval. If you are feeding results into an LLM, reranking also cuts token spend, because you send 5 precise chunks instead of 20 speculative ones. That interaction between retrieval quality and inference cost is a big theme in our LLM cost optimization playbook.

Failure modes we see in production

• The model never saw your domain. General-purpose embeddings can conflate your product names, internal jargon, or regional dialect. Symptoms: confidently wrong neighbors. Fixes: contextual chunk enrichment, hybrid weighting toward BM25, or fine-tuning an open-weight model on your query logs.
• Stale index drift. Documents update, vectors do not. Wire indexing into your content pipeline as an event-driven job, not a nightly cron you hope still runs.
• Score thresholds treated as truth. A cosine similarity of 0.82 is not "82 percent relevant." Scores are only comparable within one model and one corpus. Calibrate thresholds against labeled data, never against vibes.
• Multilingual corpora, monolingual model. French queries against English-embedded documents fail silently. Use a multilingual model or per-language indexes, and decide this before you embed a million chunks.
• No feedback loop. Without click and reformulation logging, you cannot see which queries fail, so quality plateaus wherever it launched.

Measure it or you are guessing

Search quality regressions are invisible in unit tests and brutal in production. You need an evaluation harness before you need a better model:

1. Build a golden set. Collect 100 to 300 real queries from logs (or realistic ones from domain experts) and label the relevant documents for each.
2. Pick metrics. Recall@k tells you whether the right documents entered the candidate pool. nDCG@10 and MRR tell you whether they ranked where users look.
3. Baseline BM25 first. It is your control group. If your vector stack cannot beat tuned BM25 on your golden set, do not ship it.
4. Re-run on every change. New embedding model, new chunking, new HNSW parameters, new reranker: every change goes through the harness. Treat it like CI for relevance.
5. Close the loop in production. Log queries, clicks, dwell, and zero-result rates. Feed failures back into the golden set monthly.

This is an afternoon of engineering that pays for itself the first time someone asks "did the new model make search better or worse?" and you answer with a number.

Search UX is part of the ranking function

Retrieval quality means nothing if the presentation wastes it. The details that move engagement:

• Latency budget of roughly 200 to 400 ms end to end. Users abandon slow search. Reserve most of the budget for retrieval plus rerank, and cache frequent queries.
• Honest snippets. Show the passage that actually matched, with the semantic match highlighted, not the first 160 characters of the document.
• Filters and facets still matter. Semantic search complements structured filtering, it does not replace it. Date, category, and language filters do work no embedding can.
• Handle the zero-result case. With hybrid retrieval and a sane fallback ("no exact matches, here are the closest results"), a true dead end should be rare. Log every one.
• Ask for feedback sparingly. A small "was this helpful" signal on result clicks builds the training data for your next quality jump.

A pragmatic build order

If we were starting your project on Monday, the sequence would be:

1. Ship tuned BM25 with logging first. It is the baseline and the fallback.
2. Add a hosted embedding model, structure-aware chunking, and an HNSW index behind hybrid RRF fusion.
3. Build the golden-set evaluation harness and lock it into CI.
4. Add a cross-encoder reranker over the top 100 candidates.
5. Only then consider fine-tuning, query rewriting, or exotic index tuning, and only where the metrics say the ceiling is.

Most teams invert this order, spend a quarter on model selection, and ship without evaluation or a lexical fallback. The order above ships value in week one and improves measurably every week after.

How Innovation T can help

Innovation T designs and builds retrieval systems end to end: embedding pipelines, hybrid search over Elasticsearch or dedicated vector databases, reranking layers, evaluation harnesses, and the search UX on top. We have made these tradeoffs on real corpora and real latency budgets, and we can tell you quickly which parts of the stack your product actually needs.

If search is becoming a differentiator (or a complaint) in your product, explore our software and AI engineering services or talk to our team about an architecture review.