Statistical Significance
Prompt B scored 84% vs A's 80% on 200 calls. Is B better — or did it get lucky?
Why this matters
Every evaluation score is an estimate from a finite sample. Evaluate two *identical* prompts on 100 random calls each and one will often "win" by 3–5 points purely by chance. Teams that ship the winner of every A/B comparison accumulate noise, not improvement.
show how uncertain each score is; ask whether a difference is larger than chance would produce; planning tells you how many labelled calls you need *before* running the comparison. And because both prompts usually run on the same calls, like are far more powerful than comparing two independent accuracies.
Core concept
Technical explanation + simple intuition + example
Confidence intervals and sample size
An accuracy of 84% on 200 calls really means "probably somewhere between ~78% and ~88%". Two prompts whose ranges overlap a lot might not differ at all. To shrink the range by half, you need four times as many calls.
Paired comparisons: McNemar
If both prompts ran on the same calls, most calls are easy (both right) or hopeless (both wrong). Only calls where they *disagree* tell you which is better. If B is right on 34 of those and A on 18, that lopsided split is evidence; a 26/26 split isn't.
The 10 questions
Every important concept should answer all of these
1What is it?
Tools for quantifying uncertainty in metric estimates and deciding whether differences exceed chance variation.
2Why does it exist?
Finite evaluation samples produce noisy scores; decisions based on noise waste effort and cause regressions.
3What problem does it solve?
It prevents shipping 'improvements' that are random fluctuations and sizes evaluation sets appropriately.
4How does it work internally?
Report CIs; use paired tests (McNemar, paired bootstrap) for same-item comparisons; plan sample size by power analysis.
5What does it look like on real transcript data?
Label noise, ambiguous calls and sampling variance all add noise; slices have much smaller n than the whole set.
6What can go wrong?
Declaring winners from point estimates; unpaired tests on paired data; p-hacking via many comparisons; confusing statistical and practical significance.
7How do I evaluate it?
Pre-register the comparison, metric and α; report effect size with CI and p-value; check power.
8When should I use it?
Every model/prompt comparison that informs a decision.
9When should I NOT use it?
Don't treat p < 0.05 as proof of business value; don't skip effect sizes.
10How does it affect downstream systems?
Determines which changes ship, how much evaluation data to label, and how confident reports to stakeholders can be.
Interactive lab: A/B prompt evaluation simulator
Compare two prompts: Wilson CIs, difference CI, p-value and required sample size update live. Then run McNemar on paired results, and watch two identical prompts 'beat' each other across 1,000 simulated evaluations.
Wilson CI, McNemar and paired bootstrap
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
Over six months a team shipped 14 prompt "improvements", each winning by 1–3 points on a 150-call dev set. A retrospective on a fresh 1,000-call sample showed the final prompt was no better than the version from month two. Afterwards: 600-call comparisons, paired McNemar tests, a minimum effect size of 2 points, and a monthly fresh sample.
Common mistakes
Noise produces 'winners'.
Fix: Require a significant paired test and a meaningful effect size.
Throws away the pairing's power.
Fix: McNemar or paired bootstrap when both systems ran on the same items.
Can't detect realistic improvements.
Fix: Plan n with a power calculation before labelling.
A significant 0.3-point gain may not matter.
Fix: Report effect size with a CI and judge practical value.
Engineering decision
How do you decide whether prompt B beats prompt A?
- Paired test on the same items (McNemar / paired bootstrap)
- Pre-registered metric, α and minimum effect
- Enough items per power analysis
- Point estimates alone
- Repeatedly testing on the same set until significant
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.