Classification Metrics
Precision, recall, F1, accuracy — and why the right metric is a business decision.
Why this matters
Every classification metric is a different summary of the same four numbers in the : true positives, false positives, false negatives, true negatives. Choosing which summary to optimize is choosing which mistakes you're willing to make.
For "genuinely interested customer" detection: a false positive wastes a sales call; a false negative loses a customer who wanted a loan. Which is worse depends on your costs — and that decides whether you chase , , or a weighted balance.
Core concept
Technical explanation + simple intuition + example
The four cells and the metrics built from them
- Precision: of the customers we flagged, how many were really interested?
- Recall: of the really interested customers, how many did we flag?
- F1: a single number that's high only when both are high.
- : fraction of all calls we got right — dominated by the easy majority.
Thresholds and business costs
Most classifiers output a score; you choose the cut-off. Lower threshold → catch more interested customers (recall ↑) but call more uninterested ones (precision ↓). Put a price on each mistake and the best threshold falls out of the arithmetic.
The 10 questions
Every important concept should answer all of these
1What is it?
Numerical summaries of a classifier's agreement with gold labels, derived from the confusion matrix.
2Why does it exist?
Different errors have different consequences; metrics make them visible and comparable.
3What problem does it solve?
They quantify how well a system serves its purpose and guide threshold and model choices.
4How does it work internally?
Count TP/FP/FN/TN per class; compute precision, recall, F1, specificity; sweep thresholds for curves.
5What does it look like on real transcript data?
Interested customers are a minority; accuracy flatters; costs of FP and FN differ greatly by business line.
6What can go wrong?
Optimizing accuracy on imbalanced data; confusing precision with recall; ignoring prevalence effects on precision; choosing thresholds on the test set.
7How do I evaluate it?
Per-class precision/recall/F1 with CIs, PR curves, cost at the chosen operating point.
8When should I use it?
Always for classification; choose the headline metric with stakeholders.
9When should I NOT use it?
Don't use accuracy as the headline on imbalanced tasks.
10How does it affect downstream systems?
The chosen metric drives prompt/model choices, thresholds and the business outcome (wasted calls vs lost customers).
Interactive lab: Confusion-matrix playground
Edit TP/FP/FN/TN and watch precision, recall, F1, accuracy, specificity and F-β update. Then choose a decision threshold on overlapping score distributions using business costs for false positives and false negatives.
Metrics and cost-optimal threshold in scikit-learn
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
primary_metric ("precision", "recall" or "f_beta"), beta, min_precision, min_recall. Sales has only 5 callers (wasted calls hurt), but a missed customer costs about 4× a wasted call.Real-world application
A sales team was told the lead classifier had "96% accuracy". Callers complained that most leads were cold. Precision on "interested" was 38%: only 4% of calls were genuinely interested, so the accuracy came from correctly ignoring uninterested calls. Re-framing the goal as "precision ≥ 0.7 at recall ≥ 0.5" and tuning the threshold on a validation set made the leads usable.
Common mistakes
The majority class dominates.
Fix: Report per-class precision/recall and macro-F1.
They answer different questions (flagged vs actual).
Fix: Precision: 'of what I flagged'; recall: 'of what exists'.
0.5 is arbitrary.
Fix: Pick the operating point from business costs on validation data.
Engineering decision
Which metric should you optimize for 'interested customer' detection?
- Precision when false leads are expensive (limited callers)
- Recall when missing a customer is expensive (high customer value)
- F_β / expected cost when both matter with known ratios
- Accuracy as the sole metric under imbalance
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.