Open-set vs Closed-set Classification
Consistency and governance vs discovery and flexibility — and the hybrid most teams end up with.
Why this matters
Ask "why wasn't the customer interested?" two ways. : pick from {Rate, Eligibility, Documentation, Timing, Other}. : describe the reason in your own words. Closed sets are consistent, countable and governable — but blind to anything you didn't anticipate. Open sets discover new problems ("the app keeps crashing") — but produce "Rate too high", "Pricing concern" and "High EMI" for the same thing, and different names on every run.
Most production systems need both: closed for measurement, open for discovery, and a governance loop that moves discoveries into the taxonomy.
Core concept
Technical explanation + simple intuition + example
The trade-off
- Closed set: consistent, easy to evaluate and trend; can't see new categories.
- Open set: finds the unexpected; hard to count, drifts, needs normalization.
- Hybrid: closed set + "Other: <free text>", reviewed periodically; recurring "Other" clusters get promoted into the taxonomy.
The 10 questions
Every important concept should answer all of these
1What is it?
Closed set: predict from a fixed label list. Open set: generate or discover categories not fixed in advance.
2Why does it exist?
Measurement needs stable categories; the world produces new ones.
3What problem does it solve?
Choosing well balances comparability over time with the ability to discover new issues.
4How does it work internally?
Closed: enum outputs + exact metrics. Open: free-text reasons → normalize/cluster → human review. Hybrid: both, with a promotion loop.
5What does it look like on real transcript data?
Objection reasons have a long tail; new product/app issues appear suddenly; free-text reasons vary by run and model.
6What can go wrong?
Closed sets that silently absorb new issues into 'Other'; open sets that fragment and drift; no owner for taxonomy updates.
7How do I evaluate it?
Closed: per-class metrics. Open: cluster purity against human-reviewed samples, naming stability across runs, human usefulness ratings.
8When should I use it?
Closed for KPIs and compliance; open for discovery phases; hybrid for ongoing production monitoring.
9When should I NOT use it?
Open-set outputs as direct KPIs; closed sets during early exploration of a new product.
10How does it affect downstream systems?
Determines what dashboards can show, how quickly new issues surface, and how much review effort is needed.
Interactive lab: Open vs closed set simulation
See the same 16 objections through a closed set, an open set across three runs (naming drift), and a discover-then-govern hybrid. Finish with a trade-off matrix and four scenarios.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A closed objection taxonomy tracked "Other" at a steady 8%. When a bank's mobile app broke during a release, "Other" rose to 14% — a signal nobody read, because "Other" wasn't reviewed. A weekly job now clusters "Other" free-text reasons; a spike in a coherent cluster pages the product owner. The "Digital experience" category was added (v3.2) after three weeks.
Common mistakes
New issues hide there.
Fix: Cluster and review 'Other' regularly.
Naming drift makes counts meaningless over time.
Fix: Normalize into governed categories before counting.
The taxonomy goes stale.
Fix: Version and promote recurring clusters.
Engineering decision
Closed, open or hybrid?
- Closed: KPIs, compliance, trend reporting
- Open: new products, exploratory research
- Hybrid: steady-state production with discovery
- Open for KPIs
- Closed without an 'Other' review process
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.