Prompt Engineering
From “Is this customer interested?” to a specification the model can actually follow.
Why this matters
A prompt is a specification. The model will fill every gap you leave with its own assumptions: what counts as “interested”, whether a maybe is a yes, whether the agent's words count, what format to answer in. On noisy transcripts, those assumptions are exactly where the errors live.
Good prompting is not magic phrasing. It is: a clear role, a precise task, explicit label definitions, constraints (including what NOT to do), an output format, evidence requirements and well-chosen examples — then measuring the result on a test set.
Core concept
Technical explanation + simple intuition + example
Anatomy of a production prompt
- : who the model is and the standing rules.
- Task instruction: exactly what to decide.
- Label definitions & constraints: what each label means; what to do when unsure; what is NOT evidence.
- Examples: a few worked cases, especially the hard boundaries.
- Output format: machine-readable, with the exact allowed values.
- Input: the transcript — last, so the question is fresh.
Zero-shot, few-shot and in-context learning
: instructions only. : instructions plus a handful of solved examples. Examples teach boundaries better than definitions — but they also teach accidents: if all your examples are "interested", the model leans that way.
The 10 questions
Every important concept should answer all of these
1What is it?
Designing the model's input — instructions, definitions, constraints, examples and output format — so outputs are correct, consistent and parseable.
2Why does it exist?
LLMs are general; your task has specific definitions and failure modes the model doesn't know.
3What problem does it solve?
It converts an implicit business question into an explicit, testable specification.
4How does it work internally?
Iterate: write the spec, run on a dev set, read failures, add targeted constraints/examples, re-measure. Keep a held-out test set untouched.
5What does it look like on real transcript data?
Hinglish hedges (shayad, dekhte hai), leading agent questions, sarcasm, resolved objections and diarization errors are the cases prompts must address explicitly.
6What can go wrong?
Vague labels, no abstain option, no evidence rules, few-shot label imbalance, prose outputs, prompt bloat, and overfitting prompts to the test set.
7How do I evaluate it?
Label accuracy/macro-F1, format validity, evidence verbatim-and-customer rate — on a dev set while iterating, on a test set once.
8When should I use it?
Always when using an LLM; invest most in definitions, abstain rules and evidence constraints.
9When should I NOT use it?
Don't try to fix upstream data errors (e.g., diarization) purely by prompting; don't keep adding rules without measuring.
10How does it affect downstream systems?
Prompt quality drives accuracy, parse failures, evidence quality, cost (length) and cacheability.
Interactive lab: Prompt playground
Edit the system instruction, task, constraints, output format and few-shot examples. Run on six hard test transcripts and see label accuracy, format validity and evidence quality. Use the simulator, or a real model via the backend.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A first prompt (“classify interest: yes/no”) gave 71% accuracy and a pile of parse errors. Iterations — on a dev set — added: an insufficient_evidence label with a definition (+9 pts), “only customer words are evidence” (+4 pts, and evidence audits became possible), a “final stance” rule for resolved objections (+3 pts), and JSON output with an enum (parse errors 6% → 0.2%). The team froze the prompt, evaluated once on the test set, and versioned the prompt alongside the model name.
Common mistakes
The model applies its own notion of 'interested'.
Fix: Define each label with inclusion/exclusion criteria (mirror your annotation guideline).
Forced choices turn ambiguity into confident errors.
Fix: Add insufficient_evidence with a definition and 'do not guess'.
Demonstrations bias the label distribution and teach nothing about boundaries.
Fix: Use balanced, boundary-focused examples from a training split.
The final number is inflated.
Fix: Iterate on dev; touch test once (see Gold Set Design).
Engineering decision
What should go into the prompt vs elsewhere?
- Definitions, abstain rules, evidence rules and output format → prompt
- Examples of hard boundaries → few-shot section
- Stable rules → cached system prefix
- Deterministic transformations (number parsing) → code, not prompt
- Large reference documents that are rarely relevant → retrieve or omit
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.