Hinglish ASR
Speech recognition for Hindi-English code-switching inside one sentence, tested on real names, numbers and dates rather than studio audio.
Hinglish ASR handles Hindi-English code-switching in one sentence. It needs test data with real names, numbers and dates, not just clean studio audio.
Hinglish ASR handles Hindi-English code-switching in one sentence. It needs test data with real names, numbers and dates, not just clean studio audio.
Related
The short version
- Hinglish ASR is speech recognition built for Hindi and English mixed inside a single sentence, not two systems bolted together at the language boundary.
- The hard part is not acoustics alone. It is code-switching, transliteration with no single spelling, and numbers, names and dates that change meaning when a syllable is misheard.
- Word error rate hides the failures that cost money. Measure entity accuracy, slot accuracy and task success on calls from your own phone lines.
- Studio recordings flatter every model. Evaluate on 8 kHz telephony audio with street noise, interruptions and real names.
- There is no credible single WER figure for Hinglish. Anyone quoting one without a dataset, a domain and a script policy is guessing.
- Recording calls in India needs a lawful basis under DPDP plus a retention rule, and the test set is personal data like any other.
What Hinglish ASR actually is
Hinglish ASR is automatic speech recognition that handles Hindi and English inside the same utterance, the same turn and often the same clause. A caller says something like order do din late hai, please refund process kar dijiye, and the recogniser has to produce a usable transcript without deciding that the sentence belongs to one language. The output may be Devanagari, Roman script or a mixture, and that choice is a product decision rather than a property of the model.
It is not Hindi ASR with an English word list. A bilingual lexicon helps, but the failure mode is structural: the grammar switches mid-sentence, the acoustic model hears a phoneme sequence that belongs to neither language cleanly, and the language model has to score a path that no monolingual corpus contains. It is also not translation. Translation happens after recognition, and a recogniser that silently translates is destroying the evidence you need for evaluation. And it is not code-mixing detection, which only labels the phenomenon instead of transcribing it.
The term itself is loose. Linguists distinguish code-switching, where a speaker alternates between languages, from code-mixing, where elements are embedded inside a sentence, and from borrowing, where a word has been absorbed into the host language. In an Indian contact centre all three appear in the same call, often in the same minute. Hinglish is the market name for the bundle, and it is useful precisely because it does not pretend the boundaries are clean.
One sentence, two grammars, three ambiguities
Take a sentence a real caller produces: mera payment do baar cut ho gaya, refund kab aayega. The Hindi frame carries the grammar, the English nouns payment and refund sit inside it, and the verb forms are Hindi. A recogniser trained mostly on Hindi will render payment as a Hindi word that sounds similar and lose the entity. One trained mostly on English will transcribe the Hindi as noise and hallucinate a fluent English sentence that means something else. Both produce a transcript that looks plausible and is wrong in the place that matters.
The second ambiguity is transliteration. The same name can be written Shruti, Shruthi, Sruti or Shrutii, and a downstream CRM match will fail on three of the four. The same spoken word can be rendered in Devanagari or Roman script, and a mixed transcript can switch between them mid-sentence. There is no authoritative spelling for most Indian names in Roman script, so the recogniser is not solving a recognition problem, it is making an orthographic choice on the caller behalf.
The third ambiguity is semantic and it is the dangerous one. Kal means both yesterday and tomorrow. Saal, sal and saal can be a year or a year with a different vowel length. Do can be two or the imperative give. A transcript that is phonetically correct can still invert a date or a quantity, and a downstream agent that acts on it will refund the wrong order or book the wrong slot. This is why entity accuracy matters more than word accuracy for a voice agent, and why the evaluation has to look at the slot values rather than the sentence.
The pipeline from audio to a decision
A production Hinglish pipeline has more stages than a demo. Audio arrives over a telephony codec at eight kilohertz, which removes the high frequencies that separate similar consonants. Voice activity detection cuts the stream into speech segments and has to survive street noise, a television in the background and a caller who pauses mid-sentence to think. The recogniser then produces one or more candidate transcripts, and a normalisation stage converts numbers, currency, dates and names into a canonical form before anything downstream sees them.
Each stage has its own failure mode. Aggressive voice activity detection clips the first syllable of a name. Endpointing that waits too long makes the agent feel slow, and one that cuts early splits a sentence and destroys the grammar. A recogniser with a good average word error rate can still be terrible on proper nouns, because names are rare in the training distribution and carry most of the information. Normalisation that maps das and dedh into the same bucket will produce a confident, wrong quantity.
The last stage is intent and slot extraction, and it should be evaluated separately from transcription. A transcript can be imperfect and still yield the right intent if the keyword survives. It can also be near-perfect and yield the wrong slot because a number was formatted differently. Keeping the stages separable in the logs is what lets you tell whether a failure was acoustic, lexical, orthographic or semantic, and that distinction is the difference between a fix and a guess.
Where the ambiguity actually lives
Ambiguity is not evenly distributed across a transcript. It clusters in six places, and each has a different mitigation. Names are the worst, because they are rare, personally important and impossible to spell consistently. Numbers and money come second, because lakh and crore are natural units in Indian speech and a model trained on Western numeric text will render them as unrelated digits. Dates are third, because relative expressions such as kal and parso are resolved against the call date, not the training date.
Then there are the borrowed English nouns, which are usually the easiest because they are common in both corpora, and the Hindi verb forms, which carry the intent and are acoustically compact. Addresses and landmarks are their own category: a Bengaluru caller may name a junction that no national corpus contains, and the correct transcription depends on local knowledge rather than language modelling. Product names and plan names are the sixth, and they are the ones a business cares about most because they map directly to a catalogue.
The practical mitigation is a domain lexicon that is versioned alongside the model. Add the names of your products, your cities, your competitors and your common misspellings, and weight them in decoding. A lexicon will not fix a bad acoustic model, but it will stop a recogniser from turning a plan name into a random English phrase, which is the failure that reaches the customer as a wrong answer about price.
Normalisation rules that prevent silent errors
Normalisation is where a speech system earns or loses trust, and it is usually the least glamorous part of the build. The job is to convert what was said into one canonical representation before any downstream logic runs: a number as digits, a date as an ISO value with a resolved reference day, a currency amount as an integer in paise, a name as a string with a stable spelling, a phone number as ten digits without spaces. If two stages disagree about the format, the agent will compare a string to a number and fail in a way that looks like a model problem.
The rules need to be explicit about Indian conventions. Lakh and crore are not optional local colour; they are how people quote money and volume, and a system that cannot parse pandrah lakh will fail on a routine property enquiry. Dedh, sava, paune and saadhe are fractions in common use and should map to 1.5, 1.25, 0.75 and 1.5 with the noun determining the unit. A time of eleven thirty should be unambiguous about morning and evening, and a relative date should be resolved at the moment of the call and stored with that reference, not re-resolved later.
Write the rules as a table with an input pattern, an output format and a test case, and run the table as part of the build. The table is readable by a operations lead who does not write code, which matters because the person who knows that callers say nikaal instead of nikal should be able to add a rule without a sprint. Keep the raw transcript alongside the normalised value so that a dispute can be resolved by listening to the audio rather than trusting either representation.
Why clean audio flatters every model
A demo recorded on a laptop in a quiet room tells you almost nothing about a call from a shop floor in Ludhiana. Production audio is eight kilohertz, compressed, clipped, and often arrives with a fan, a television, a second conversation and a horn. Speakers move away from the microphone, cover it with a hand, or switch to speakerphone in a moving auto. Every one of those conditions costs accuracy, and the loss is not uniform: it hits consonants, names and numbers first, which are exactly the tokens the agent needs.
There is also the accent problem, which is really a corpus problem. Hindi spoken by a Tamil-first speaker, a Bengali-first speaker and a Marathi-first speaker differs in vowel colour and stress, and a corpus dominated by one region will underperform on the others. Add English words pronounced with Indian stress patterns and you get a distribution that no single public dataset represents well. This is not a reason to abandon public data; it is a reason to treat it as a starting point and to measure on your own traffic.
The honest position is that no single number describes Hinglish ASR quality. A model can score well on read speech, poorly on telephony, and very differently on names versus function words. The only defensible claim is a measured result on a stated dataset, in a stated domain, with a stated script policy. Anything else is a marketing sentence, and it will not survive the first week of real calls.
Evaluation that matches the job
Word error rate is the standard first metric and it is not enough. It treats every word as equally important, so it punishes a missing article as hard as a wrong amount, and it says nothing about whether the agent completed the task. On a Hinglish call, a transcript that drops please and sir but keeps the order number and the amount is a success. A transcript that is word-perfect except for one digit in a phone number is a failure. WER cannot tell those apart, which is why it should be reported next to metrics that can.
Four measures cover most of what matters. Entity error rate looks only at the named entities and values, and is the closest proxy for business risk. Slot accuracy compares the extracted intent arguments with a human-labelled truth, including the null case where the caller did not provide a value. Task success is the end-to-end outcome, judged by whether the agent did the right thing. False accept rate counts cases where the agent proceeded confidently on a wrong value, which is worse than asking again.
Report every metric by slice: language mix, acoustic condition, speaker gender, region and call duration. An overall number hides the slice that is failing, and the failing slice is usually the one with the least training data. Keep a fixed evaluation set and a rotating fresh set, because a model tuned against the fixed set will improve on paper while the production experience stays flat. The fresh set is the only defence against that.
Building a test set from your own calls
Start with consent. Recording a call in India requires a lawful basis under the DPDP Act, notice that is understandable, and a retention rule that covers the audio, the transcript and the derived labels. A test set is not exempt because it is internal. If you cannot lawfully keep a recording, you cannot keep it in an evaluation corpus, and a vendor that offers to build one from scraped audio is offering you a liability rather than a dataset.
Sample for coverage rather than convenience. Pull calls across the hours you actually operate, the languages you actually support, the products you actually sell and the outcomes you actually see, including the abandoned ones. Thirty calls from a single enthusiastic agent on a quiet Tuesday will produce a test set that flatters the model and misses the calls that generate complaints. Label the truth by listening, not by reading the transcript, because reading the transcript anchors the labeller to the model output.
Keep the set small enough to maintain and large enough to detect a regression. A few hundred carefully labelled utterances, refreshed quarterly, will catch most changes that matter, and it will stay in human review range. Version it, hash it, and record which model version was scored against it. When someone asks whether the new release is better, the answer should be a diff on a fixed set with the audio available, not a recollection that it felt smoother in the demo.
Streaming, endpointing and the barge-in problem
A voice agent cannot wait for the end of a paragraph. It has to transcribe as audio arrives, decide when the caller has finished, and respond before the silence becomes uncomfortable. That constraint changes the recognition problem: a streaming model has less right context, so its accuracy on the final word of a segment is lower, and endpointing errors become recognition errors. Cut too early and you lose the verb; wait too long and the caller repeats themselves, which produces overlapping audio that is even harder.
Barge-in adds a second stream. The caller interrupts the agent mid-sentence, and the system has to stop speaking, keep the caller audio, discard the agent echo and resume recognition without losing the thread. Echo cancellation is doing the heavy lifting, and a cheap headset will beat a clever model. The design rule is that the agent should stop on the first meaningful syllable rather than on any noise, and that a false stop is much cheaper than talking over a customer who is trying to correct an amount.
The practical decision is where to spend latency. A two-pass design recognises quickly for the endpoint decision and then re-scores the full segment with more context before extracting slots, which costs a few hundred milliseconds and buys a much better number. For a voice agent billed per connected minute at a published rate plus eighteen percent GST, the cost of a re-score is trivial next to the cost of a wrong refund. Measure the p95 rather than the average, because the callers who wait are the ones who complain.
Worked example: qualifying a Bengaluru property enquiry
A real-estate team runs an inbound voice agent for projects in north Bengaluru. Callers switch between Kannada, Hindi and English, and the agent has to capture budget, configuration, locality and timeline before it books a site visit. The first version was evaluated on read speech and looked excellent. On real calls it failed in three specific ways: it misheard lakh as lack, it normalised dedh crore as fifteen crore, and it could not spell the locality names consistently enough to match the inventory database.
The fixes were unglamorous. A domain lexicon added the project names, the localities and the common mispronunciations. A normalisation rule mapped dedh, sava and paune to fractions with an explicit unit, and required the currency before any amount reached the CRM. A confirmation step read the captured value back to the caller, so a wrong number became a question rather than a booking. And the evaluation set was rebuilt from two hundred consented calls across the three languages, labelled by two people and scored on slots rather than words.
The result was not a perfect transcript. It was a system that caught its own uncertainty: when the confidence on an amount was low, it asked again instead of guessing, and the site visit booked with a wrong budget stopped appearing. That is the realistic goal for Hinglish ASR in production. You are not trying to match a human transcriber. You are trying to make the expensive errors visible before they become actions.
Frequently asked questions
Is Hinglish ASR just Hindi ASR with English words added?
No. Adding English vocabulary helps with borrowed nouns, but the failure is grammatical: the sentence switches structure mid-clause, and a monolingual language model assigns low probability to the correct path. You need training or adaptation data that contains real code-switched utterances, and an evaluation set that reflects how your callers actually speak rather than how a newsreader reads.
What word error rate should we expect?
There is no honest general number. WER depends on the dataset, the domain, the acoustic condition, the script policy and whether names are counted. Any vendor quoting a single figure without those details is giving you a marketing number. Ask for a result on a sample of your own audio, scored on entities and slots as well as words, and compare two systems on the same sample.
Should the transcript be in Devanagari or Roman script?
It depends on what consumes it. Devanagari is more consistent for Hindi grammar and easier to search in Hindi, while Roman script matches how most urban users type and how CRM fields are usually stored. Many teams keep both: a canonical Devanagari transcript for the record and a Roman normalised form for matching. What matters is that the choice is explicit and applied consistently, because mixed output breaks string matching.
How do we handle names that have no fixed spelling?
Store a canonical spelling per contact and match on a phonetic key rather than the raw string. Maintain a lexicon of the names and places that matter to your business, and let an operator add variants when a real call exposes one. Do not expect the recogniser to solve orthography; treat the spelling as a data decision downstream of recognition, and keep the audio so a disputed name can be checked.
Why does the model fail on numbers and amounts?
Indian numeric speech uses units and fractions that are rare in the corpora most models are trained on: lakh, crore, dedh, sava, paune. A model may transcribe the words correctly and still format them wrongly, or hear a similar-sounding word and produce a plausible number. Normalise explicitly, require the currency before an amount is used, and read high-value values back to the caller before acting.
Do we need a custom model?
Often not at first. A strong multilingual model plus a domain lexicon and a solid normalisation layer will handle a large share of traffic. Custom adaptation earns its cost when the domain has unusual names, a strong regional accent mix or a vocabulary that the base model consistently mangles. Measure the gap on your own labelled calls before funding a training run, because the operational work is usually the bigger win.
How does DPDP affect call recording for evaluation?
A recording is personal data, so you need a lawful basis, clear notice at the start of the call, a stated purpose and a retention window. Consent for service delivery is not automatically consent to keep audio for model training. Keep the evaluation corpus minimal, restrict access, hash identifiers where possible, and include the corpus in erasure workflows so a data principal request can actually be honoured.
What is the most useful single metric?
Slot accuracy on the fields that trigger an action, such as amount, date, order number and phone number. It maps directly to business risk, it is easy to explain to a non-technical stakeholder, and it exposes the failures that a good word error rate hides. Pair it with false accept rate, because a system that guesses confidently is worse than one that asks a second question.
Can we evaluate without our own labelled calls?
You can start with a public corpus, and you should, because it gives you a baseline before you have traffic. But public Hinglish data rarely matches your domain, your codec or your callers, so the baseline will overstate quality. Treat it as a smoke test and build a small consented set from your own calls as soon as you have the lawful basis to keep one.