Error Analysis
Overall F1 0.81 — but 0.62 on noisy transcripts. Slicing and root-cause analysis turn metrics into a plan.
Why this matters
An aggregate metric is an average over very different situations. Clean English inbound calls and very noisy Romanized-Hindi outbound calls can differ by 30 points of F1 — and the aggregate hides which one you're failing.
slices performance by metadata (agent, call type, language, quality, duration, segment, model, prompt version), reads failures, forms hypotheses and tests them. The discipline is separating a *symptom* ("Agent B's calls fail") from a *root cause* ("Agent B takes noisier calls").
Core concept
Technical explanation + simple intuition + example
Slicing
Compute the metric separately for each group: per agent, per language, per transcript quality… The slices that are much worse than average — and big enough to matter — are where to look.
Root cause vs confounders
Agent B looks bad. But if Agent B mostly handles noisy outbound calls, the real problem may be audio quality, not the agent. Cross the two dimensions: if within each quality tier all agents score the same, quality is the cause and "agent" was a confounder.
The 10 questions
Every important concept should answer all of these
1What is it?
Systematically finding where and why a system fails by slicing metrics, reading errors and testing hypotheses.
2Why does it exist?
Aggregates hide failures; engineering effort needs a target.
3What problem does it solve?
It converts 'the model is 81%' into 'fix X for slice Y, worth Z points of business impact'.
4How does it work internally?
Slice by metadata, rank by gap × volume, cross-tab to control confounders, read and categorize failures, test fixes on the slice.
5What does it look like on real transcript data?
Transcript quality, language mix, agent, call type and duration are strong drivers; they're correlated with each other.
6What can go wrong?
Blaming confounded proxies (agents), chasing tiny slices, multiple-comparison false alarms, fixing symptoms not causes.
7How do I evaluate it?
Per-slice metrics with CIs; before/after comparisons on the target slice with paired tests.
8When should I use it?
After every evaluation round, and whenever aggregate metrics move.
9When should I NOT use it?
Don't over-interpret slices with a handful of examples.
10How does it affect downstream systems?
Drives the roadmap: ASR work, prompt changes, routing rules, human-review targeting.
Interactive lab: Error-analysis dashboard
900 labelled predictions with rich metadata. Filter, slice by any dimension (with bootstrap CIs), cross-tabulate two dimensions, and answer the root-cause question about Agent B.
Slice report with pandas
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
After a model upgrade, overall F1 rose 2 points and the team celebrated. Slicing showed F1 on Romanized-Hindi calls had *dropped* 6 points while English rose 4 — a real regression for 20% of customers, hidden by the mix. They shipped only after adding Romanized-Hindi few-shot examples and verifying the slice recovered.
Common mistakes
Slices can regress while the aggregate improves.
Fix: Standard slice report with every evaluation.
Correlated dimensions confound each other.
Fix: Cross-tabulate or model adjusted effects.
Noise.
Fix: Show n and CIs; confirm on fresh data.
Numbers say where, not why.
Fix: Read and categorize 20–50 failures per suspect slice.
Engineering decision
Which slice should you work on first?
- Large gap × large traffic share × fixable cause
- Slices tied to business risk (compliance, high-value customers)
- Tiny slices with wide CIs
- Proxies that are confounded by another dimension
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.