Sampling Parameters
Temperature, top-p and max tokens: why the same prompt can give different answers.
Why this matters
Run the same classification prompt twice and you may get “interested” then “insufficient_evidence”. That isn't the model changing its mind — it's sampling. Each output token is drawn from a probability distribution, and the decoding parameters reshape that distribution.
For extraction and classification, variability is noise you pay for in inconsistent labels and flaky evaluations. For brainstorming, it's a feature. Knowing exactly what , and do lets you choose deliberately.
Core concept
Technical explanation + simple intuition + example
Temperature
Think of temperature as changing how strongly the model prefers its most likely next token. Low temperature: “stick to your favourite”. High temperature: “give the runners-up a real chance”.
Top-p (nucleus) and max tokens
Top-p keeps only the most likely tokens that together make up p of the probability (e.g., 90%) and throws away the long tail of weird options. Max tokens is a hard cap on output length — hit it and the answer stops mid-sentence, or mid-JSON.
Determinism in practice
Temperature 0 means “always take the top token” — so outputs should be identical. In real APIs they're *mostly* identical but not guaranteed, because the numbers computed on shared hardware can differ slightly from run to run.
The 10 questions
Every important concept should answer all of these
1What is it?
Decoding parameters that control how tokens are chosen from the model's next-token distribution and how long generation runs.
2Why does it exist?
LLMs define distributions, not single answers; decoding turns distributions into text.
3What problem does it solve?
They let you trade diversity for consistency and bound cost/length.
4How does it work internally?
Temperature rescales logits before softmax; top-p truncates to a probability nucleus; max_tokens caps length.
5What does it look like on real transcript data?
Ambiguous calls (hedges, sarcasm) have flat label distributions — exactly where sampling produces inconsistent labels.
6What can go wrong?
High temperature → inconsistent labels and fabrications; low max_tokens → truncated JSON; assuming T = 0 is perfectly deterministic.
7How do I evaluate it?
Run each input several times; measure label agreement across runs, unique outputs, and truncation (stop reason) rates.
8When should I use it?
Low temperature for classification/extraction; higher for ideation or when sampling multiple candidates deliberately (self-consistency).
9When should I NOT use it?
Don't raise temperature to 'fix' a wrong answer; don't set tight max_tokens with long structured outputs.
10How does it affect downstream systems?
Sampling variance shows up as evaluation noise, flaky regressions and inconsistent decisions for the same call.
Interactive lab: Sampling laboratory
Move temperature and top-p and watch a next-token distribution reshape. Generate repeated outputs from a tiny language model and measure diversity and truncation. Optionally repeat a real model call 5× at your temperature.
Temperature and top-p on a toy distribution
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
An evaluation dashboard showed prompt v7 beating v6 by 1.5 points — then losing by 1 point the next day, with nothing changed. Both ran at temperature 0.7. Re-running at temperature 0 with three repeats per call showed the two versions were indistinguishable; the day-to-day swings were sampling noise on the ~12% of genuinely ambiguous calls.
Common mistakes
Labels flip on ambiguous inputs; evaluations become noisy.
Fix: Use T = 0 or low; measure run-to-run agreement.
It reshapes the entire distribution, including the label decision.
Fix: Think in probabilities: which tokens gain mass?
Truncated outputs get parsed as if complete.
Fix: Check the stop reason; size max_tokens with headroom.
Engineering decision
What decoding settings should a transcript classifier use?
- T = 0 (or ≤ 0.2) for classification and extraction
- Moderate T with multiple samples for self-consistency confidence estimates
- Top-p ~0.9–1.0 when sampling; max_tokens sized for your longest valid output
- High temperature for single-shot decisions
- Tight max_tokens with verbose reasoning fields
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.