Multi-stage Pipeline Design
Sequential, parallel, routed: decomposition buys debuggability — and costs latency, money and compounding errors.
Why this matters
One giant prompt that does everything is simple to build and impossible to debug. A of focused stages — normalize, extract, classify, collect evidence, verify — lets you measure and improve each part separately.
But every stage adds latency and cost, and errors compound: five stages at 95% each are ≈77% end-to-end if errors are independent (and correlated errors on noisy calls make it worse). Good pipeline design uses parallelism for independent stages, to spend effort where it matters, and verification to catch errors before decisions.
Core concept
Technical explanation + simple intuition + example
Sequential, parallel and routed topologies
- Sequential (A → B → C → D): each step uses the previous result. Simple, but latency adds up.
- Parallel (A → {B, C, E} → D): independent steps run at the same time; latency is set by the slowest branch.
- Routing (Input → Router → Simple / Complex / Ambiguous): cheap path for easy cases, expensive path or humans for hard ones.
Decomposition and error propagation
Splitting a task makes each piece easier and measurable — but each handoff is a place for errors to enter and travel downstream. Decompose where it buys you measurability or accuracy; don't add stages because it “feels” more rigorous.
The 10 questions
Every important concept should answer all of these
1What is it?
Designing a system as a graph of components (code, models, LLM calls, humans) with explicit inputs/outputs.
2Why does it exist?
Complex tasks mix skills; one monolithic call is hard to debug, evaluate and optimize.
3What problem does it solve?
It enables per-stage evaluation, targeted fixes, parallelism, routing and cost control.
4How does it work internally?
Define stage contracts (schemas), wire a DAG, parallelize independent stages, route by difficulty/confidence, verify before decisions, log everything.
5What does it look like on real transcript data?
Noisy calls fail several stages at once (correlated errors); most calls are easy and could use a cheap path.
6What can go wrong?
Latency explosion from long sequential chains, compounding errors, stages that discard needed context, routers that misroute, verifiers that rubber-stamp.
7How do I evaluate it?
Per-stage metrics on gold stage outputs, end-to-end metrics by slice, latency p50/p95, cost per decision, review rates.
8When should I use it?
When sub-tasks are separable and measurable, or when routing can save cost on easy inputs.
9When should I NOT use it?
When a single well-grounded call meets requirements; when the latency budget can't absorb extra stages.
10How does it affect downstream systems?
Topology determines where errors enter, how fast decisions are made, and how much each decision costs.
Interactive lab: Architecture builder
Load sequential, parallel and routing presets or wire your own pipeline on a canvas. 2,000 simulated transcripts flow through it to estimate end-to-end accuracy, errors caught, p50/p95 latency, cost and LLM calls per transcript.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A seven-stage pipeline (each stage an LLM call) hit 11 seconds p95 and $9 per 1k calls. Analysis showed three stages were independent (NER, objections, evidence) and one (normalization) could be deterministic code. After parallelizing and replacing the LLM normalizer, p95 fell to 4 seconds and cost to $5.40 — with identical accuracy. Adding a router that sent clean, short calls to a small model cut cost further to $3.10 with a 0.4-point macro-F1 loss, which the business accepted.
Common mistakes
Latency adds up for no benefit.
Fix: Identify independent stages; run them concurrently.
Adds cost, latency and variance.
Fix: Deterministic steps (number parsing, schema checks) in code.
Errors correlate with hard inputs.
Fix: Measure end-to-end on slices.
Misrouted hard cases get cheap, wrong answers.
Fix: Evaluate the router like any classifier; monitor per-route quality.
Engineering decision
Single call or multi-stage pipeline?
- You need per-stage measurability (speaker, entities, intent, evidence)
- Some sub-steps are deterministic or reusable
- Input difficulty varies enough to justify routing
- A single grounded call meets quality and latency targets
- Extra stages can't be evaluated on their own gold labels
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.