Context Engineering
What goes into the context window, in what order — and why more context can make answers worse.
Why this matters
Modern models accept huge , so it's tempting to stuff in everything: the full rulebook, old calls, the product catalogue, policy PDFs. Every token costs money on every call, adds latency, and gives the model more places to look in the wrong direction.
is deciding what the model sees: relevant rules and examples, the transcript, pre-extracted evidence, the question — ordered so stable parts are cacheable and critical parts aren't buried.
Core concept
Technical explanation + simple intuition + example
Budget, relevance and order
Treat the context window like a briefing document for a busy expert. Include what they need, leave out what they don't, put the standing instructions first and the actual question last. Irrelevant pages don't just waste money — they distract.
Position effects: lost in the middle
Models tend to use information at the very beginning and very end of a long context more reliably than information in the middle. The effect varies a lot between models and has shrunk in newer ones — which is exactly why you measure it on yours instead of assuming.
Retrieval vs stuffing
Stuffing = put everything in every call. Retrieval = pick the relevant pieces per call. Stuffing is simple and never misses a document; retrieval is cheaper and more focused but can miss what matters.
The 10 questions
Every important concept should answer all of these
1What is it?
The design of everything placed in a model's input: instructions, knowledge, examples, conversation state, retrieved evidence and the query.
2Why does it exist?
Output quality, cost and latency all depend on what the model sees and where.
3What problem does it solve?
It prevents prompt bloat, distraction and position effects while enabling caching.
4How does it work internally?
Budget tokens per block, include only relevant content, order static→dynamic, pre-extract evidence, measure position sensitivity.
5What does it look like on real transcript data?
Long calls (30+ minutes), long rulebooks, repeated history from previous calls, and multiple products per call.
6What can go wrong?
Context overflow, bloat, buried evidence, conflicting duplicated rules (v1 and v2 rulebook), dynamic content breaking the cache prefix.
7How do I evaluate it?
Ablate blocks (with/without), vary positions, track cost and latency per call, and measure accuracy on long-call slices.
8When should I use it?
Every LLM system; it matters most for long inputs and high volume.
9When should I NOT use it?
Don't micro-optimize token counts on low-volume prototypes before measuring quality.
10How does it affect downstream systems?
Context design sets cost per decision, latency, cache hit rate and accuracy on long calls.
Interactive lab: Context-window simulator
Toggle and reorder context blocks, watch token usage, cost, cacheable prefix and warnings update. Then explore the beginning/middle/end position experiment — simulated, or for real against a model through the backend.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team concatenated the last three calls with each new call “for context”. Accuracy on the current call's intent dropped 4 points: the model sometimes answered based on an older call where the customer had been interested. Cost rose 2.7×. Replacing history with a two-line structured summary (“previous outcome: not interested, reason: rate”) placed after the current transcript restored accuracy and cut cost.
Common mistakes
Irrelevant tokens cost money and distract.
Fix: Add blocks only when an ablation shows they help.
The prefix differs every call, so nothing is cached.
Fix: Static first: system, rules, examples; then transcript and query.
Old and new rulebooks both present → inconsistent behaviour.
Fix: Version and deduplicate prompt components.
They differ across models and prompts.
Fix: Run a position experiment on your model.
Engineering decision
Stuff it or retrieve it?
- Stuff: small, stable, always-relevant knowledge (label definitions)
- Retrieve: large, changing, mostly irrelevant-per-call corpora
- Select segments: very long calls
- Stuffing large corpora into every call
- Retrieval when the answer is always in the transcript
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.