Taxonomy Design
Labels are a product: one dimension per field, mutually exclusive siblings, an honest 'other'.
Why this matters
A is the vocabulary your whole system speaks: what the prompt asks for, what annotators label, what dashboards count. A flawed taxonomy can't be fixed downstream — overlapping categories cap agreement, missing categories force wrong labels, mixed dimensions make metrics meaningless.
Good taxonomies are designed deliberately: separate dimensions (product, intent, objection) into separate fields, make siblings , keep depth shallow enough to label reliably, and give every set an explicit abstain/other option.
Core concept
Technical explanation + simple intuition + example
Dimensions, exclusivity and exhaustiveness
Ask of every label set: "Is this one question?" Product (which loan?), intent (do they want it?) and objection (why not?) are three questions — three fields. Within a field, each call should fit exactly one label (exclusive), and every call should fit some label (exhaustive — which is what "Other"/"Insufficient evidence" is for).
Depth, granularity and governance
Deeper trees look precise but each leaf gets fewer examples and more disagreement. Keep leaves you can label reliably and act on. Treat the taxonomy like an API: versioned, owned, with definitions, and changed deliberately.
The 10 questions
Every important concept should answer all of these
1What is it?
The designed set of categories (and their relations) that a system uses to label content.
2Why does it exist?
Every downstream artifact — prompts, guidelines, gold labels, metrics, dashboards — inherits the taxonomy.
3What problem does it solve?
It makes labels reliable, actionable and comparable over time.
4How does it work internally?
Separate dimensions into fields; make siblings exclusive; add abstain/other; keep depth shallow; test coverage on real data; version and govern.
5What does it look like on real transcript data?
Calls mix products and intents; objections have a long tail; new issues (app outages) appear without warning.
6What can go wrong?
Mixed dimensions, overlapping siblings, missing abstain, overly deep leaves, unused labels, silent changes.
7How do I evaluate it?
Coverage tests (fits none/several), inter-annotator agreement per label, label frequency, confusion between siblings.
8When should I use it?
Before annotation, prompting or dashboard design — and on a regular review cadence.
9When should I NOT use it?
Don't over-design for categories that never occur or that nobody acts on.
10How does it affect downstream systems?
Taxonomy flaws show up as low κ, noisy metrics, confused prompts and misleading dashboards.
Interactive lab: Taxonomy builder
Load a bad taxonomy (mixed dimensions, overlapping siblings, too deep) and fix it with drag-and-drop until the linter is happy. Then test coverage: can every real utterance get exactly one leaf?
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
product (array of allowed values, must include 'unspecified'), intent (must include 'insufficient_evidence'), objection (must include 'other'). Each list should have 3–6 values.Real-world application
An objection dashboard showed "Pricing issue" falling 30% after a relaunch — celebrated as a pricing win. In reality, annotators had started using the newly added "EMI too high" label for the same calls. The underlying objection rate hadn't moved. After merging the overlapping labels (with a sub-reason field) and back-mapping history, the trend was flat.
Common mistakes
A call can legitimately have several labels; annotators split arbitrarily.
Fix: One field per dimension.
Two labels describe the same thing → disagreement and fake trends.
Fix: Merge; use sub-reasons if needed.
Gaps force wrong labels.
Fix: Add an explicit abstain/other with a definition.
Trends break silently.
Fix: Version and back-map.
Engineering decision
Should you add, split or merge a category?
- Add: a recurring, actionable 'Other' cluster
- Split: a category whose sub-types need different actions and can be labelled reliably
- Merge: siblings with persistent disagreement or no distinct action
- Adding categories for rare, non-actionable cases
- Changing definitions without versioning
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.
Advanced challenge
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.