Text Normalization
Making noisy transcripts consistent — without destroying the signal you need.
Why this matters
ASR output for the same sentence varies wildly: personal lon, persnal loan, PL; chahiye, chaiye, chahie; 5 lakh, five lac, 50 L (which is ten times more!). Keyword rules, fuzzy matching, entity extraction and even LLM prompts behave more predictably on consistent input.
But is lossy by design. Every step deletes a distinction: lowercasing merges "US" and "us"; stripping punctuation turns "10.5" into "10 5"; stop-word removal deletes "nahi" and flips the meaning. The engineering skill is choosing which distinctions you can afford to lose — and keeping the original text for evidence.
Core concept
Technical explanation + simple intuition + example
The normalization ladder
Think of normalization as a ladder from "exactly what was said" to "what it means":
- Basic: case, whitespace, repeated punctuation.
- Spelling: fix known misspellings (
lon → loan). - : pick one spelling per Hindi word (
chaiye → chahiye). - Semantic: numbers, abbreviations, canonical phrases (
five lac → 500000,PL → personal loan).
Each rung up makes matching easier and throws more information away.
When normalization destroys information
A normalizer is a compression algorithm with no undo button. If a distinction matters to any downstream consumer — negation for intent, "?" for dialogue acts, exact numbers for extraction, casing for names — normalizing it away creates errors nobody can recover later.
The 10 questions
Every important concept should answer all of these
1What is it?
Transformations that map many surface variants of text to a canonical form: case, whitespace, punctuation, spelling, transliteration, abbreviations, numbers, and sometimes meaning-level frames.
2Why does it exist?
ASR and informal writing produce unbounded variation. Downstream matching, counting and extraction need consistency.
3What problem does it solve?
It reduces sparsity: fewer distinct strings for the same thing, so rules, dictionaries, fuzzy matchers and classifiers generalize better.
4How does it work internally?
A pipeline of ordered functions — regexes, lookup tables, transliteration maps, number grammars, and optionally learned models — each with a clear input/output contract.
5What does it look like on real transcript data?
Hinglish transcripts mix scripts and spellings (chahiye/chaiye), write numbers as words (five lac), use abbreviations (PL, EMI) and repeat fillers (haan haan). Most of these are enumerable; some (sarcasm, hedging) are not normalizable at all.
6What can go wrong?
Destroying negation, decimals, ranges, question marks, names and emphasis; wrong expansions of ambiguous abbreviations; order-dependent bugs; losing the ability to quote the original evidence.
7How do I evaluate it?
Build a labelled set of (raw, expected normalized) pairs including adversarial cases; measure exact-match accuracy per rule. More importantly, measure downstream F1 with and without each step (ablation).
8When should I use it?
For deterministic, well-specified variation (numbers, units, known spellings) and before matching/counting. Keep it conservative before an LLM — LLMs already handle much surface variation.
9When should I NOT use it?
Don't apply aggressive steps (stop-words, punctuation stripping) before intent classification; don't normalize the text you'll quote as evidence.
10How does it affect downstream systems?
Normalization choices change what NER can find, what fuzzy matching matches, token counts and cost, and whether evidence quotes can be verified against the original transcript.
Interactive lab: Progressive normalization playground
Toggle normalization stages on a noisy utterance, see exactly what each step changed, then run a battery of information-preservation tests against your pipeline.
A non-destructive normalizer in Python
Real Python, running in your browser (Pyodide)
Try it yourself
Conceptual (A), applied (B) and engineering (C) exercises
mujhe abhi loan nahi chahiye?50 L. Normalize it to an integer amount in rupees.Graded by a transparent keyword rubric (not an LLM) — it checks for the key ideas, then shows a model answer for self-assessment.
Real-world application
A team added "standard NLP preprocessing" (lowercase, strip punctuation, remove stop-words) before their intent classifier. Accuracy on the aggregate dropped only 2 points, so it shipped. Error analysis later showed that recall on *Not Interested* had collapsed on Hinglish calls: "nahi chahiye" had become "chahiye". The aggregate hid it because refusals were a minority class.
The fix: remove stop-word removal entirely, keep punctuation, run number normalization into a separate structured field, and quote evidence from the original text.
Common mistakes
Stop-word lists and punctuation stripping were designed for bag-of-words search, not intent or evidence tasks.
Fix: Start from zero steps; add each step only when an ablation shows a downstream gain.
Stripping punctuation before number parsing breaks decimals and ranges; correcting spelling before protecting names corrupts them.
Fix: Parse numbers and protect entities first; write order-sensitive test cases.
You can no longer quote verbatim evidence or debug what the customer actually said.
Fix: Store original + normalized text + an offset map.
“PL” is personal loan in retail calls and profit & loss in business calls.
Fix: Expand only unambiguous abbreviations, or let the model interpret them in context.
Engineering decision
How much normalization should come before an LLM?
- The variation is enumerable and the mapping is exact (numbers, units, known misspellings of product names)
- You need structured, comparable fields (amount in ₹) for analytics
- Fuzzy matching or keyword rules run downstream
- The downstream model is an LLM that already handles spelling variation — aggressive steps only remove signal
- The text will be quoted as evidence
- A step touches negation, punctuation or casing that carries meaning
Quiz
Pass mark 60%. Missed questions go to your mistake log and spaced review.
chahiye, chaiye, chahie, chahye are best described as…rate 10.5 percent?PL ka statement bhej do. Your normalizer expands PL → personal loan. What's the best fix?Advanced challenge
budget 5-6 lac ka hai, 3 saal ke liye as JSON with fields original (unchanged string), amount_min, amount_max (integers in ₹) and tenure_months (integer).