Embeddings and Semantic Similarity
“borrow money” and “personal loan” share no words — embeddings still put them side by side.
Why this matters
Lexical methods compare spellings. Customers don't use your vocabulary: they say "EMI bahut zyada hai" when your taxonomy says "rate / affordability", or "I need some money urgently" when you're looking for "personal loan". map text to vectors so that similar *meanings* land close together, enabling , clustering and deduplication.
But similarity is a property of the model, not the text. English-only models handle Romanized Hindi poorly, and negations ("I don't want a loan") often sit right next to their positive versions. You must evaluate embeddings on your own data.
Core concept
Technical explanation + simple intuition + example
Vectors and cosine similarity
An embedding model turns a sentence into a list of a few hundred numbers — a point in space. Sentences with similar meaning end up pointing in similar directions. measures that angle: 1 = same direction, 0 = unrelated.
Lexical vs semantic similarity
(character or word overlap) is cheap, exact and great for typos and IDs, but blind to synonyms. Neural embeddings capture paraphrase but can blur exact distinctions and negation. Many production systems use both (hybrid search).
The 10 questions
Every important concept should answer all of these
1What is it?
Dense vector representations of text in which semantic similarity corresponds to geometric closeness.
2Why does it exist?
Meaning-level matching is needed when wording varies (paraphrase, synonyms, code-mixing).
3What problem does it solve?
It enables semantic search, clustering, deduplication and similarity-based features.
4How does it work internally?
Encode texts with a pretrained sentence encoder; compare with cosine similarity; index with approximate nearest neighbours for scale.
5What does it look like on real transcript data?
Hinglish and Romanized Hindi challenge English-only models; ASR noise perturbs embeddings; short utterances ('haan') carry little meaning alone.
6What can go wrong?
Negation blindness, topic-over-stance similarity, poor multilingual coverage, thresholds copied across models, over-trusting 2D plots.
7How do I evaluate it?
Labelled pairs (similar/not) → AUC or accuracy at threshold; retrieval recall@k on real queries; per-language slices.
8When should I use it?
Semantic search, clustering objections, finding similar calls, retrieving few-shot examples, deduplication.
9When should I NOT use it?
Exact lookups (IDs, rates by key), decisions that hinge on negation, or as the sole classifier without evaluation.
10How does it affect downstream systems?
Embedding choices shape clusters (and thus taxonomies), retrieval quality in RAG, and similarity thresholds.
Interactive lab: Embedding explorer
Enter sentences and compare a lexical TF-IDF baseline with a real neural embedding model (all-MiniLM-L6-v2) running in your browser. See a 2D PCA map, a cosine-similarity heatmap and semantic search.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team used an English sentence-transformer to deduplicate objection reasons. Precision looked fine on English; on Romanized-Hindi reasons, unrelated objections ("CIBIL kam hai" and "documents nahi hai") were merged because the model mostly saw shared function words. Switching to a multilingual model and validating on 300 labelled Hinglish pairs raised pair-classification AUC from 0.71 to 0.88.
Common mistakes
Negations and opposite stances can be close.
Fix: Evaluate on pairs that differ in stance; don't use similarity alone for decisions.
Poor coverage of Romanized Hindi.
Fix: Use multilingual models and validate per language.
Similarity scales differ by model.
Fix: Re-tune thresholds on labelled pairs.
Projection distorts distances.
Fix: Use plots for intuition; decide with measured similarities.
Engineering decision
Lexical, dense or hybrid?
- Lexical: exact terms, IDs, typos
- Dense: paraphrase-heavy text, clustering by meaning
- Hybrid: search over real user language
- Dense-only when exact terms matter (product codes)
- Lexical-only when users paraphrase heavily
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.