Inter-Annotator Agreement
Raw agreement ≠ agreement beyond chance: Cohen's κ, Fleiss' κ and Krippendorff's α.
Why this matters
Before you measure a model against labels, measure the labels against themselves. tells you how reproducible your ground truth is — an upper bound on how precisely any system can be evaluated.
The trap: raw agreement looks great when one label dominates. Two annotators who both say "Interested" 90% of the time will agree ~82% of the time *by chance alone*. Chance-corrected statistics — , , — ask the right question: how much better than chance are we?
Core concept
Technical explanation + simple intuition + example
Cohen's kappa: agreement beyond chance
Kappa compares how often annotators agree (pₒ) with how often they'd agree by guessing with their usual label frequencies (pₑ). κ = 1 is perfect, κ = 0 is chance-level, negative is worse than chance.
More than two annotators, and missing labels
Fleiss' κ extends the idea to many annotators (each item rated by the same number of people). Krippendorff's α is the most flexible: any number of annotators, missing labels allowed, and it supports ordinal/interval data with a distance function.
The 10 questions
Every important concept should answer all of these
1What is it?
Statistics measuring how consistently multiple annotators label the same items, corrected for chance agreement.
2Why does it exist?
Labels are the reference for all metrics; unreliable labels make metrics unreliable.
3What problem does it solve?
It quantifies label reliability, reveals ambiguous categories, and sets realistic ceilings for model performance.
4How does it work internally?
Double-label a sample, compute κ/α, inspect the confusion between annotators, revise guidelines, repeat.
5What does it look like on real transcript data?
Skewed label distributions (most calls not interested) make raw agreement misleadingly high; hedges and sarcasm drive disagreements.
6What can go wrong?
Reporting raw agreement only; ignoring prevalence effects; computing κ on a non-representative sample; using Fleiss with missing labels.
7How do I evaluate it?
Report κ/α with raw agreement, marginals, sample size and a CI (bootstrap); per-category agreement.
8When should I use it?
Whenever you create or buy labels — pilots, ongoing QA (e.g., 10–15% double-labelled).
9When should I NOT use it?
Don't compare κ across datasets with very different prevalence without caution.
10How does it affect downstream systems?
Low agreement caps achievable metrics, makes model comparisons noisy, and indicates taxonomy/guideline problems.
Interactive lab: Agreement lab
Three annotators label twenty utterances. Change any label and watch raw agreement, Cohen's κ (with its pₒ/pₑ breakdown), Fleiss' κ and Krippendorff's α update. Try the majority-class paradox preset.
κ and α in Python
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A labelling vendor reported "93% agreement" on compliance-violation labels. The violation rate was 4%. Computing κ gave 0.31: annotators agreed mostly on the easy "no violation" calls and rarely on which calls contained violations. The client renegotiated the contract to specify κ ≥ 0.7 on a stratified sample containing at least 30% violations.
Common mistakes
Chance agreement inflates it under class imbalance.
Fix: Report κ/α alongside raw agreement and marginals.
Random samples of imbalanced data are mostly easy negatives.
Fix: Stratify the agreement sample to include rare classes.
κ depends on prevalence and number of categories.
Fix: Use bands as rough guidance; compare like with like.
Items with missing ratings must be dropped.
Fix: Use Krippendorff's α for incomplete designs.
Engineering decision
Which agreement statistic should you report?
- Cohen's κ: exactly two annotators
- Fleiss' κ: fixed number (>2) of raters per item
- Krippendorff's α: missing labels, varying raters, ordinal data
- Raw agreement alone on imbalanced labels
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.