Text Classification
Multi-class, multi-label, hierarchical — the label structure you choose decides what you can measure.
Why this matters
“Is this customer interested?” sounds like one question. In real calls it's several: interested in *which* product, *existing* customer or new, *needs follow-up* or not. Forcing all of that into one label list — or splitting it wrongly — creates label disagreements, uninterpretable confusion matrices and metrics that can't be compared.
The structure of your labels — , or — is an engineering decision made *before* any model is trained or prompted.
Core concept
Technical explanation + simple intuition + example
Three label structures
- Multi-class: pick exactly one (Interested / Not Interested / Unknown).
- Multi-label: pick any number (Interested + Existing Customer + Needs Follow-up).
- Hierarchical: pick a path in a tree (Loan → Personal Loan → Interested), and back off to a parent when unsure.
Why taxonomy design creates evaluation problems
If labels overlap (“Interested” vs “High Intent”), two annotators can both be right and still disagree. If there's no “Unknown” label, ambiguous calls get forced into a guess. If one list mixes product, intent and status, the confusion matrix mixes apples and oranges — and no metric on it means anything.
The 10 questions
Every important concept should answer all of these
1What is it?
Assigning predefined category labels to text — one label, several, or a path in a hierarchy.
2Why does it exist?
Business processes route, count and act on categories.
3What problem does it solve?
It turns each call into decisions and aggregates (conversion funnels, objection rates, compliance flags).
4How does it work internally?
Rules, classical ML (TF-IDF + logistic regression), fine-tuned transformers, or LLM prompts with constrained label sets.
5What does it look like on real transcript data?
Calls contain multiple products and facts, hedged language, objections that get resolved, and third parties — all of which stress single-label schemes.
6What can go wrong?
Overlapping labels, missing abstain class, mixed dimensions, labels that never occur, and choosing multi-class when facts co-occur.
7How do I evaluate it?
Multi-class: confusion matrix, per-class P/R/F1, macro-F1. Multi-label: per-label P/R, Hamming loss, subset accuracy. Hierarchical: per-level metrics. Always alongside inter-annotator agreement.
8When should I use it?
Multi-class for one mutually exclusive dimension; multi-label for co-occurring facts in one dimension; hierarchical when categories nest and back-off is useful.
9When should I NOT use it?
Don't use one multi-class list for multiple dimensions; don't use hierarchies deeper than your data supports.
10How does it affect downstream systems?
Label structure fixes the output schema, the annotation guideline, the metrics, and what dashboards can show.
Interactive lab: Label-structure designer
Label the same snippets as multi-class and multi-label and compare the metrics. Then design your own label structure and stress-test it on real snippets with a taxonomy linter.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
intent (one of interested/not_interested/insufficient_evidence), customer_status (new/existing), products_of_interest (array).Real-world application
A team tracked “Interested / Not Interested / Callback / Existing customer” as one multi-class label. Inter-annotator κ was 0.41 and never improved with training. Splitting into three fields — intent {interested, not_interested, insufficient_evidence}, customer_status {new, existing}, next_action {callback, none} — raised κ to 0.78 on the same calls without changing the annotators.
Common mistakes
Items legitimately belong to several labels; annotators and models disagree arbitrarily.
Fix: One field per dimension; mutually exclusive values within each.
Ambiguous calls are forced into a guess, inflating majority classes.
Fix: Add an explicit abstain value with a definition.
Subset accuracy is harsh and accuracy is ill-defined.
Fix: Report per-label P/R/F1 and Hamming loss.
Engineering decision
Multi-class, multi-label or hierarchical?
- Multi-class: exactly one answer per call on one dimension
- Multi-label: several co-occurring facts on the same dimension (objection types)
- Hierarchical: nested categories and you need to back off when unsure
- Multi-class across dimensions
- Deep hierarchies with sparse leaves
- Multi-label lists that mix dimensions — use separate fields
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.
Advanced challenge
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.