Tokenization
What the model actually reads — and why Hinglish transcripts can cost more to process.
Why this matters
Every LLM call is priced, rate-limited and length-limited in , not words. When you process 100,000 call transcripts a month, the difference between 1.3 and 2.5 tokens per word is the difference between a budget that works and one that doesn't.
Tokenization also decides what the model "sees". A misspelling like lon or a Romanized Hindi word like chahiye is split into fragments the model must reassemble. That affects cost, usage, latency, and how you long calls.
Core concept
Technical explanation + simple intuition + example
Words vs tokens
A tokenizer is a fixed dictionary of text pieces. Common pieces (" loan", " the") are single entries. Rare pieces get built from smaller ones, like spelling a rare word with Lego bricks.
So "tokens per word" is not a constant: it's low for frequent English and higher for rare spellings, other scripts, numbers and noise.
Subword algorithms: BPE and SentencePiece
BPE learns its vocabulary by repeatedly gluing together the most common adjacent pair. After enough merges, frequent words become single tokens and rare words stay split.
SentencePiece is a toolkit that does this (or a related "Unigram" method) directly on raw text including spaces, so it doesn't need a language-specific word splitter.
Token count → cost, context, latency, chunking
Tokens are the unit of everything operational: - Cost: you pay per input and output token. - Context: the model can only hold so many tokens at once. - Latency: more input tokens → longer time to first token. - Chunking: if a call is too long, you split it by tokens — and a naive split can cut a number like "5 lakh" in half.
The 10 questions
Every important concept should answer all of these
1What is it?
Tokenization converts text into a sequence of integer IDs from a fixed vocabulary of subword units. Tokens, not words, are what the model processes.
2Why does it exist?
Neural models need a finite input alphabet. Word-level vocabularies can't handle unseen words; character-level sequences are too long. Subword tokenization is the compromise.
3What problem does it solve?
It solves the open-vocabulary problem: any string — misspellings, new product names, Hinglish — can be represented without an 'unknown' token.
4How does it work internally?
A pre-tokenization regex splits text; a learned merge table (BPE) or likelihood-pruned vocabulary (Unigram) maps each chunk's bytes to the longest/most-probable known pieces, which are looked up to IDs.
5What does it look like on real transcript data?
Noisy transcripts contain misspellings (lon), Romanized Hindi (chahiye, kitna), repeated fillers (haan haan) and spelled-out numbers (five lac). These typically produce more tokens per word than clean English — measure it in the lab.
6What can go wrong?
Budget estimates based on word counts; context overflow on long calls; naive chunking that splits numbers or turns; assuming token counts are identical across providers; assuming a fixed multilingual 'penalty'.
7How do I evaluate it?
Measure tokens per call on a representative sample (by language and quality tier) with the tokenizer of the model you'll use. Track p50/p95 tokens per call, not just the mean.
8When should I use it?
Always count tokens with the target model's tokenizer when estimating cost, setting max input length, designing chunking, or choosing a model with enough context.
9When should I NOT use it?
Don't use one provider's tokenizer to budget another provider's model, and don't treat token counts as a measure of 'information content'.
10How does it affect downstream systems?
Token counts drive cost per decision, the choice between single-call and multi-stage pipelines, whether you need chunking/retrieval, and caching strategy (static prefix size).
Interactive lab: Interactive tokenizer
Type or paste any text. See the real tokens, their IDs, the count and an estimated cost — then compare English, Hindi, Hinglish, Romanized Hindi and a noisy transcript side by side.
Count tokens and cost in Python
Real Python, running in your browser (Pyodide)
This uses a simple whitespace/regex split as a stand-in for a real tokenizer (tiktoken isn't available in the browser Python runtime). The pattern — measure a sample, then budget on percentiles — is what matters.
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
persnal lon produce more tokens than personal loan, even though it has fewer characters?Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A collections team wants to classify 200,000 calls/month. Their first estimate used "average 800 words per call × $X per 1k words". Measured with the model's tokenizer, noisy Hinglish calls averaged far more tokens per word than their English test set, and the p95 call was 4× the median. The budget and the context-window plan both had to change.
The fix wasn't a new model: they (1) measured tokens per call by language slice, (2) trimmed the rulebook and put it in a cached prefix, and (3) chunked long calls by conversation segment instead of fixed token windows.
Common mistakes
Tokens per word varies widely with language, script, spelling and numbers.
Fix: Tokenize a stratified sample with the target model's tokenizer and budget on p95, not the mean.
Each model family has its own vocabulary; the same transcript can differ substantially in token count.
Fix: Use each provider's token-counting tool/endpoint when comparing models on cost.
Cutting every N tokens splits numbers (5 la|kh), turns and question→answer pairs.
Fix: Chunk on turn/segment boundaries within a token budget, with overlap.
The ratio depends on the tokenizer, the spelling conventions and the content. It changes when the provider updates its tokenizer.
Fix: Measure periodically and store token counts per call in your logs.
Engineering decision
When does tokenization need explicit engineering attention?
- Estimating cost or rate limits for high-volume transcript processing
- Calls can approach the model's (long calls + rulebook + examples)
- Designing chunking or segment selection for long transcripts
- Comparing providers — count with each provider's tokenizer
- Low-volume prototypes where cost is negligible — just log token usage
- Using token counts as a proxy for meaning or quality
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.
Advanced challenge
Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.