Prompt Caching and Batching
Stop paying full price to re-read the same 5,000-token rulebook 100,000 times.
Why this matters
In a transcript pipeline, most of each prompt is identical across calls: the system prompt, the labelling rulebook, the few-shot examples. Only the transcript changes. Without , you pay to process that static prefix on every single call — often 70–80% of your input bill.
attacks cost from the other side: if results can wait (nightly analytics), asynchronous batch processing typically trades hours of latency for a substantial discount and higher throughput. Both are architecture decisions, not afterthoughts.
Core concept
Technical explanation + simple intuition + example
How prompt caching works
The model processes your prompt left to right. If the beginning of today's prompt is byte-for-byte identical to a recent one, the provider can reuse the already-computed work for that prefix instead of recomputing it. You pay a reduced rate for the reused part — and the answer starts sooner.
Batching: latency for price and throughput
Instead of sending 200,000 requests one by one and waiting for each, you submit them as one job and collect results later. The provider can schedule them efficiently, so it's cheaper — but you might wait hours.
The 10 questions
Every important concept should answer all of these
1What is it?
Prompt caching reuses processed prompt prefixes across requests; batching groups requests for efficiency (server-side) or discounted async processing (batch APIs).
2Why does it exist?
Transcript pipelines repeat large static prompts and often don't need instant answers.
3What problem does it solve?
They cut cost (and, for caching, latency) without changing model quality.
4How does it work internally?
Put static content first and keep it byte-identical; monitor cache hit rates; route offline workloads to batch processing.
5What does it look like on real transcript data?
Rulebooks and examples are large and stable; transcripts vary; nightly analytics can wait; live assist can't.
6What can go wrong?
Dynamic content (timestamps, customer names) at the start of the prompt breaks caching; cache expiry between sparse calls lowers hit rates; batch jobs used for latency-sensitive paths.
7How do I evaluate it?
Track cached vs uncached input tokens, hit rate, cost per call and TTFT before/after; batch job completion times.
8When should I use it?
Caching: any repeated prefix of meaningful size. Batching: offline, independent requests.
9When should I NOT use it?
Batch APIs for real-time needs; caching tiny or constantly changing prompts.
10How does it affect downstream systems?
Caching changes the economics of long rulebooks and few-shot examples; batching changes when results are available.
Interactive lab: Cost simulator
Enter call volume, rulebook and conversation sizes, prices, cache multipliers, hit rate and batch discount. Compare no caching, caching, batching and both — and see what happens if the static content isn't first.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
[call ID + date] [transcript] [rulebook, 5k tokens] [4 examples] [task + output format].Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team enabled prompt caching but saw almost no savings. Their prompt began with “Today is {date}. Call ID {id}.” — unique per call, so no prefix ever matched. Moving call metadata after the rulebook raised the cache hit rate from ~0% to 93% and cut monthly input cost by 68% with identical outputs.
Common mistakes
The prefix differs every time; nothing is reused.
Fix: Static content first; dynamic content last.
Any change invalidates the cache (and changes behaviour).
Fix: Version prompts; change them deliberately.
Completion windows are long.
Fix: Batch only offline, independent work.
Engineering decision
Should this workload use caching, batching, both or neither?
- Caching: large static prefix reused across many calls within the cache lifetime
- Batching: results can wait; requests are independent
- Both: nightly analytics with a shared rulebook
- Batching for real-time
- Caching when prompts are small or unique per call
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.