Topic Modeling and Clustering
Let the data propose categories — then decide, as a human, which ones deserve to exist.
Why this matters
Your taxonomy only contains categories someone imagined. and look at thousands of customer utterances and group the similar ones, revealing themes you didn't anticipate — "the app keeps crashing", "CIBIL score worries", "wants to wait until after Diwali".
Clusters are hypotheses, not facts. The number of clusters, the embedding model and the algorithm all change the answer. The engineering skill is using clusters to *discover* and then *governing* what enters the taxonomy.
Core concept
Technical explanation + simple intuition + example
Embed → reduce → cluster → describe
- Turn each utterance into a vector (embeddings).
- Squash the vectors into fewer dimensions so clustering works better.
- Group nearby points.
- Describe each group with its most distinctive words — then a human names it.
Choosing k and judging clusters
There's no single correct number of clusters. Scores like the help, but the real test is: can a person read each cluster, give it a clear name, and would the business act on it?
The 10 questions
Every important concept should answer all of these
1What is it?
Unsupervised grouping of texts by similarity (clustering) and discovering latent themes with descriptive terms (topic modeling).
2Why does it exist?
Categories you haven't imagined can't be in your taxonomy; data can suggest them.
3What problem does it solve?
It surfaces new themes, sizes them, and gives candidate names — the input to taxonomy evolution.
4How does it work internally?
Embed, reduce dimensions, cluster (k-means/HDBSCAN), describe with c-TF-IDF keywords and examples, review with humans.
5What does it look like on real transcript data?
Short, noisy, code-mixed utterances cluster on surface features; a few themes dominate while many are long-tail.
6What can go wrong?
Arbitrary k, unstable clusters, clusters formed by language/ASR artifacts, over-trusting auto-generated names, treating clusters as ground truth.
7How do I evaluate it?
Silhouette and stability for structure; purity/NMI against a labelled sample; human coherence and actionability ratings.
8When should I use it?
Exploring 'Other' reasons, launching new products, periodic taxonomy review.
9When should I NOT use it?
As a production classifier or KPI source without review; with tiny data.
10How does it affect downstream systems?
Clusters feed taxonomy changes, guideline updates and new gold-set categories.
Interactive lab: Clustering laboratory
Embed 26 objection snippets (TF-IDF or a real in-browser MiniLM model), project with PCA, cluster with k-means, read c-TF-IDF keywords, compare silhouette across k, name the clusters and reveal the hidden themes.
Mini BERTopic-style pipeline (TF-IDF + KMeans + c-TF-IDF)
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A quarterly review embedded 40,000 "Other" objection reasons with a multilingual encoder and ran BERTopic. Of 31 topics, 9 were artifacts (greetings, ASR garbage, language-specific fillers), 14 mapped to existing categories (label leakage into 'Other'), and 8 were new. Two of those — "loan app login failures" and "waiting for festival bonus" — were large and actionable and became taxonomy v4 categories after human review.
Common mistakes
Different settings produce different clusters.
Fix: Treat clusters as hypotheses; validate with humans and labels.
Unstable, arbitrary.
Fix: Compare k with silhouette and stability; prefer interpretable, actionable clusters.
Fillers, language and ASR errors dominate lexical vectors.
Fix: Use good multilingual embeddings; filter fillers; inspect examples.
Names can be misleading.
Fix: Humans name clusters from keywords + examples.
Engineering decision
When should clustering drive a taxonomy change?
- A cluster is coherent (humans agree on a name), sizeable, stable across runs, and actionable
- Clusters driven by artifacts
- Tiny or unstable clusters
- No owner to govern the change
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.