Annotation Guidelines
The codebook decides your labels, your agreement, and therefore every metric you'll ever report.
Why this matters
Every metric compares predictions to labels. If two careful people would label the same call differently, your "ground truth" is partly noise — and no model can be measured more precisely than the labels allow.
An (codebook) turns vague business intuitions ("interested customers") into operational rules: definitions, inclusion and exclusion criteria, positive/negative/borderline examples, rulings on known edge cases, and an escalation path. It is the most leveraged document in an ML project.
Core concept
Technical explanation + simple intuition + example
Anatomy of a label definition
For each label, write: - Definition: one or two precise sentences. - Include: what counts. - Exclude: what looks similar but doesn't count. - Examples: a clear positive, a clear negative, and a borderline case with its ruling.
Edge cases and escalation
Most disagreement comes from a handful of situations: conditional interest, the agent putting words in the customer's mouth, objections that get resolved, third-party interest, sarcasm, wrong numbers, do-not-call requests. Decide them once, in writing, and annotators stop guessing.
The 10 questions
Every important concept should answer all of these
1What is it?
A written specification of how to assign labels: definitions, criteria, examples, edge-case rulings and escalation procedures.
2Why does it exist?
Humans disagree on ambiguous categories unless the decision rules are explicit.
3What problem does it solve?
It makes labels consistent (high agreement), reproducible and aligned with the business meaning.
4How does it work internally?
Draft from business goals, pilot on real data, measure agreement, study disagreements, add rulings, version, repeat.
5What does it look like on real transcript data?
Hinglish hedges, leading agent questions, sarcasm and third parties are recurring sources of disagreement.
6What can go wrong?
Vague definitions, overlapping labels, missing abstain option, examples that are all easy, undocumented rule changes.
7How do I evaluate it?
Inter-annotator agreement (κ/α) before and after revisions; disagreement analysis by category.
8When should I use it?
Before collecting any labels — gold sets, training data, LLM prompts (definitions should match).
9When should I NOT use it?
Never skip it; for tiny pilots, a one-page version still helps.
10How does it affect downstream systems?
Guidelines define gold labels, prompt definitions, metrics interpretability and dispute handling.
Interactive lab: Codebook builder
Build a mini codebook: definitions, inclusion/exclusion, examples, edge-case rulings and escalation. A reviewer checks completeness and a simulated annotator pair estimates agreement on eight tricky utterances. Download it as Markdown.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A vendor delivered 5,000 "intent" labels with κ = 0.44 between annotators. The codebook was one sentence per label. Two weeks of pilot rounds added 14 edge-case rulings (hedges, callbacks, third parties, leading questions). κ rose to 0.79, and — importantly — the model's measured accuracy "improved" by 6 points without any model change: much of the old "error" had been label noise.
Common mistakes
Annotators fill gaps with personal judgement.
Fix: Definition + include/exclude + examples per label.
Ambiguous items get forced labels, adding noise.
Fix: Add INSUFFICIENT_EVIDENCE with a definition.
Old and new labels become incomparable.
Fix: Version the codebook; record the version per label.
Easy examples teach nothing about boundaries.
Fix: Anchor every boundary with a borderline example and its ruling.
Engineering decision
How detailed should the guideline be?
- Detailed rulings when κ is below target or labels drive money/compliance
- Pilot rounds before large-scale annotation
- Shared definitions between human annotators and LLM prompts
- Adding rules for cases that never occur — keep it usable
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.