Model Selection
Quality, cost, latency, context, reliability — pick the model that satisfies the constraints, then optimize.
Why this matters
“Use the best model” is not an engineering decision. The best model for a 1-second live agent-assist budget is not the best model for a nightly batch of 3 million calls, and neither is the right one when a regulator forbids sending data outside the bank.
Model selection is constrained optimization: first eliminate models that violate hard constraints (latency, context, data residency, quality floor), then optimize the objective (cost or quality) among what's left — using numbers measured on *your* data.
Core concept
Technical explanation + simple intuition + example
The dimensions that matter
- Quality on your gold set (not a public leaderboard).
- Cost per call = input tokens × input price + output tokens × output price.
- : p50 and especially p95.
- Context length: does your longest call + prompt fit?
- Reliability: valid structured output rate, uptime, rate limits.
- Deployment: external API vs on-prem (data residency).
Dense vs mixture-of-experts, big vs small
A uses all its parameters for every token. A model has many “expert” blocks but routes each token through only a few, so it can be large in knowledge but cheaper per token to run. Small models are fast and cheap; large ones handle hard cases better.
The 10 questions
Every important concept should answer all of these
1What is it?
Choosing which model(s) serve each workload given quality, cost, latency, context, reliability and deployment constraints.
2Why does it exist?
No model is best on every dimension; workloads have different constraints.
3What problem does it solve?
It avoids overpaying, missing latency budgets, or violating data policies while meeting quality needs.
4How does it work internally?
Define constraints and objective; benchmark candidates on your gold set (quality, validity, latency, tokens); eliminate infeasible; optimize; consider routing.
5What does it look like on real transcript data?
Hinglish tokenization changes cost per call across providers; noisy calls may need larger models; long disputed calls need long context.
6What can go wrong?
Choosing by leaderboard; ignoring p95 latency; ignoring token-count differences between tokenizers; forgetting rate limits; silent model updates.
7How do I evaluate it?
Per-model: macro-F1 with CIs, JSON validity rate, p50/p95 latency, tokens and cost per call — all on the same stratified sample.
8When should I use it?
At design time and whenever prices, models or workloads change.
9When should I NOT use it?
Don't switch models on a single anecdotal failure; measure on the gold set.
10How does it affect downstream systems?
Model choice fixes cost per decision, latency budget, need for routing/verification and data governance.
Interactive lab: Model-selection simulator
Five fictional models, four workloads with hard constraints. Choose a model and justify it; then see monthly cost, feasibility against each constraint and the reasoning. Also explore latency vs throughput under batching.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team used their strongest model for everything: live assist, nightly analytics and disputes. Live assist missed its 1-second budget 30% of the time, and the analytics bill was 6× forecast. They split by workload: a small fast model for assist (with a narrower task), a mid-size MoE model for batch (quality floor met, 80% cheaper), and the strongest long-context model only for the few hundred monthly disputes.
Common mistakes
Leaderboards don't measure Hinglish call transcripts with your labels.
Fix: Benchmark on your stratified gold set.
Averages hide the slow tail users experience.
Fix: Set budgets on p95/p99.
Tokenizers differ; tokens per call differ.
Fix: Compare cost per call (or per decision) on real transcripts.
Constraints differ radically across workloads.
Fix: Select per workload; consider routing.
Engineering decision
How do you pick a model for a workload?
- Eliminate by hard constraints (latency, context, residency, quality floor)
- Then optimize the stated objective
- Consider routing when difficulty varies across inputs
- Optimizing quality when the objective is cost (or vice versa) without saying so
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.