Per-Stage Evaluation
End-to-end accuracy tells you THAT the system fails. Per-stage metrics tell you WHERE.
Why this matters
Your pipeline is 76% accurate end to end. Is the problem speaker attribution, entity extraction, the classifier, evidence selection or verification? The end-to-end number can't say — and three very different pipelines can produce the same one.
gives each stage its own gold labels and metrics, measured both in isolation (with correct inputs) and in the pipeline (with real upstream outputs). Oracle ablations then answer the engineering question directly: which stage, if fixed, buys the most?
Core concept
Technical explanation + simple intuition + example
Isolated, in-pipeline and oracle metrics
- Isolated: give the stage perfect inputs — how good is the stage itself?
- In-pipeline: give it real upstream outputs — how good is it in practice?
- Oracle ablation: replace one stage with perfect (gold) outputs — how much would end-to-end improve if that stage were perfect?
The 10 questions
Every important concept should answer all of these
1What is it?
Measuring each pipeline component against its own gold labels, in isolation and in context, plus oracle ablations.
2Why does it exist?
End-to-end metrics can't localize failures; many stage configurations give the same aggregate.
3What problem does it solve?
It directs engineering effort to the stage with the largest end-to-end impact.
4How does it work internally?
Label per-stage gold outputs on a sample, compute isolated and in-pipeline metrics, run oracle ablations, attribute each error to its first failing stage.
5What does it look like on real transcript data?
Noisy calls hurt several stages at once; diarization errors look like classifier errors downstream.
6What can go wrong?
Only labelling final outputs; comparing stages evaluated on different samples; ignoring correlated errors; optimizing a stage whose improvement doesn't reach the output.
7How do I evaluate it?
A per-stage table (isolated, in-pipeline, oracle-gain) sliced by transcript quality and language.
8When should I use it?
Any multi-stage pipeline, before deciding what to improve.
9When should I NOT use it?
Single-call systems still benefit from decomposed labels (e.g., evidence quality separate from label accuracy).
10How does it affect downstream systems?
It determines roadmap priorities and prevents 'improving' stages that don't matter.
Interactive lab: End-to-end evaluation dashboard
Three mystery pipelines with similar end-to-end accuracy. Guess which stage to fix from the headline number, then reveal per-stage metrics, oracle ablations, error attribution and failure traces.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team spent a month improving their intent prompt (+4 points isolated accuracy) and saw almost no end-to-end gain. Per-stage labelling on 300 calls showed 60% of end-to-end errors originated in speaker attribution on mono recordings. An oracle ablation confirmed it: perfect speakers would add 11 points end to end. Switching to stereo recording where available and adding a text-based role check delivered +8 points in two weeks.
Common mistakes
You can't attribute errors to stages.
Fix: Label intermediate outputs on a shared sample.
Its gains may not reach the output.
Fix: Run oracle ablations; fix the largest gain first.
Differences reflect data, not stages.
Fix: Use one sample with all stage labels.
Engineering decision
What should you fix first?
- The stage with the largest oracle-ablation gain
- The stage where most errors originate (first failing stage)
- Especially on the slices that matter most
- Choosing by isolated accuracy alone — a mediocre stage whose errors are absorbed downstream may matter little
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.