LLM-as-a-Judge
A scalable measurement instrument with known biases — validate it before you trust it.
Why this matters
Humans can't grade 100,000 reasoning strings a week. An can — cheaply and consistently. But a judge is a *measurement instrument*, and instruments have biases: it may prefer the first answer shown, the longer answer, answers written by its own model family, and it may share the generator's blind spots.
If you optimize your system against an unvalidated judge, you optimize for the judge's biases. This lesson is about using judges the way scientists use instruments: calibrate against a reference, measure the biases, and keep auditing.
Core concept
Technical explanation + simple intuition + example
Known judge biases
- : prefers whichever answer is shown first (or second).
- : prefers longer, more confident-sounding answers.
- : rates its own family's outputs higher.
- Correlated errors: can't catch mistakes it would make itself (e.g., Hinglish sarcasm).
- Judge disagreement: two judges often disagree — scores depend on the instrument.
Using judges responsibly
Give the judge a rubric and a reference answer, evaluate both orders, score components separately (label, evidence, reasoning), use a different model family where possible, and compare the judge with careful human labels on a sample before trusting it.
The 10 questions
Every important concept should answer all of these
1What is it?
Using an LLM to score or compare outputs according to a rubric or reference.
2Why does it exist?
Human evaluation doesn't scale to large volumes or rapid iteration.
3What problem does it solve?
It provides fast, cheap, consistent-ish quality signals for open-ended outputs.
4How does it work internally?
Prompt a judge model with the input, candidate output(s), rubric and optionally a reference; parse a score or preference.
5What does it look like on real transcript data?
Judges often share generators' weaknesses on Hinglish, sarcasm and hedges; they may reward confident English prose over terse correct labels.
6What can go wrong?
Position, verbosity and self-preference biases; correlated errors; drift when the judge model changes; Goodhart effects when you optimize against it.
7How do I evaluate it?
Judge–human agreement (κ) on a stratified sample; flip rate under order swap; score–length correlation; per-slice agreement.
8When should I use it?
Open-ended quality (reasoning faithfulness, summary quality) at scale — after validation.
9When should I NOT use it?
When a deterministic reference comparison exists (exact labels, fields); as the only source of truth.
10How does it affect downstream systems?
Judge scores often gate releases and prompt changes; biased judges push systems toward verbose, judge-pleasing outputs.
Interactive lab: Judge simulator
Be the human judge on eight answers, then compare with an LLM judge and the reference. Explore position bias (order swaps), verbosity bias (score vs length) and self-preference/correlated errors with interactive simulations.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team used a judge to pick between two prompt versions. The judge preferred version B in 68% of pairs. Swapping presentation order dropped that to 51%: most of B's “advantage” was position bias, because B had always been shown first. After evaluating both orders and validating against 200 human judgments (κ = 0.63 overall, 0.31 on Hinglish sarcasm), they used the judge for English reasoning quality only and kept humans on the sarcasm slice.
Common mistakes
The judge is a biased instrument.
Fix: Validate against human labels; report κ.
Position bias masquerades as a preference.
Fix: Judge both orders; count consistent wins only.
Self-preference and shared blind spots inflate scores.
Fix: Use a different family or measure the bias.
Adds noise and cost to something you can compute exactly.
Fix: Compare to gold labels directly.
Engineering decision
Should I use an LLM judge?
- Open-ended outputs without a single correct reference
- You have human labels to validate the judge
- You'll audit periodically
- Exact-match evaluation is possible
- No validation data exists
- The judge shares known blind spots with the generator on the slice you care about
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.