Class Imbalance
At 1% prevalence, a 95%-accurate detector can be wrong about most of the calls it flags.
Why this matters
Rare events are where the value is: compliance violations, fraud signals, high-intent leads on cold outbound lists. They're also where metrics lie. A model that always says “no violation” scores 99% accuracy at 1% prevalence. A genuinely good detector with 95% sensitivity and 95% specificity has only ~16% precision at that .
changes how you read metrics, how you sample evaluation data, and how you set thresholds.
Core concept
Technical explanation + simple intuition + example
Base rates drive precision
When positives are rare, even a small false-positive rate applied to the huge negative pool produces more false alarms than there are real positives. So precision collapses — even though the model itself didn't get worse.
Stratified sampling for evaluation
If you label 200 random calls at 1% prevalence, you'll get about 2 positives — sometimes zero. You can't measure recall on two examples. Stratified sampling deliberately samples more of the rare class (e.g., 100 positives + 100 negatives), then re-weights to estimate real-world numbers.
The 10 questions
Every important concept should answer all of these
1What is it?
Skewed label distributions where some classes are much rarer than others.
2Why does it exist?
Real-world events of interest (violations, hot leads) are rare.
3What problem does it solve?
Understanding imbalance prevents misleading metrics, bad thresholds and uninformative evaluation samples.
4How does it work internally?
Report prevalence-aware metrics (precision/recall/PR curves), stratify evaluation samples, re-weight to population, set thresholds by cost.
5What does it look like on real transcript data?
Outbound calls: few interested; compliance: violations in a few percent; objection types: long tail.
6What can go wrong?
Accuracy headlines, random eval samples with no positives, thresholds tuned on balanced data used on skewed traffic, forgetting to re-weight stratified estimates.
7How do I evaluate it?
Per-class precision/recall with CIs, PR-AUC, and the trivial baselines (all-negative accuracy).
8When should I use it?
Any time a class you care about is under ~20% of traffic.
9When should I NOT use it?
Balanced tasks don't need special sampling — but still report per-class metrics.
10How does it affect downstream systems?
Prevalence shifts (new campaigns, new markets) silently change precision in production.
Interactive lab: Rare-event simulator
Switch the positive rate between 1%, 5%, 20% and 50% for the same classifier and watch accuracy, precision, recall and the confusion matrix. Then see why random evaluation samples fail at low prevalence and stratified ones don't.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A mis-selling detector validated on a balanced set (50% positives) showed precision 0.92. In production, where 2% of calls were mis-sold, reviewers found only 1 in 5 flags was real. Nothing had broken — precision had been measured at the wrong prevalence. The team re-estimated precision on a score-stratified production sample, re-weighted to true prevalence, raised the threshold, and staffed review for the resulting volume.
Common mistakes
Precision depends on prevalence.
Fix: Estimate precision at production prevalence (re-weight stratified samples).
Too few positives to measure recall.
Fix: Stratify; oversample the rare class.
All-negative accuracy is 1 − prevalence.
Fix: Always report it next to accuracy.
Engineering decision
How should you evaluate a rare-event detector?
- Stratified gold sets with enough positives
- Precision/recall/PR curves at production prevalence
- Cost-based thresholds
- Accuracy as the headline
- Balanced-set precision as a production estimate
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.