How Hindi, Marathi and Hinglish agents score on task success, escalation correctness, consent capture and p95 latency, failures included.
Voice quality is not one number. A voice agent can sound natural and still fail the task, miss a consent step, or escalate at the wrong moment.
Our benchmark scores four things: task success (did the caller get what they needed?), escalation correctness (was a human brought in when required?), consent capture (was the notice read and recorded?), and latency (p50 and p95, not a single best case).
On a 150-task set across Hindi, Marathi and Hinglish, the biggest failure mode was not accent — it was turn-taking on code-mixed numbers, dates and names. Agents that asked for confirmation before booking performed better than agents that guessed.
We will publish the full method, the task set and the failures. A benchmark without failures is a brochure.
What this means for your team
Every action has a trace with tool, latency and cost.
Every consent has a notice version and a timestamp.
Every escalation has a named human and an audit entry.
Every number has a denominator and a window.
If a number in this article is not defensible, it does not ship. That is the editorial standard.
A voice agent can sound excellent and still fail the caller. Quality is a task outcome, not a tone of voice.
Measure four things separately: task success, escalation correctness, consent capture and latency at p50 and p95.
Word error rate and mean opinion score are useful diagnostics and poor buying criteria. Neither predicts whether a booking completes.
Vendor-published numbers are hard to compare because the task sets, audio conditions and normalisation rules differ, and failures are rarely published.
Build a small task set from your own recordings, score it with two raters and adjudicate the disagreements.
We publish a 420 ms median latency for voice in the India region. That is our number, on our stack, and it is not a comparison with anyone else.
One score is a marketing artefact
Voice demos are designed to be listened to, not measured. A thirty-second clip in a quiet room with a cooperative speaker will make almost any modern stack sound natural. The same stack, on a narrowband phone line, with a caller who switches from Hindi to English mid-sentence and gives a pincode twice, behaves very differently. The gap between the clip and the call is where voice projects succeed or fail, and it cannot be summarised in one number.
The industry has two familiar numbers and both are easy to misuse. Word error rate measures transcription accuracy against a reference. Mean opinion score measures how natural a listener finds the audio. Neither tells you whether the caller got their appointment, whether the agent asked for consent before recording, or whether a human was brought in at the right moment. They are diagnostics, not outcomes, and a procurement decision made on either one is a decision made on the wrong evidence.
Figure 1. A voice call is four systems in series. The caller experiences the sum, which is why a single component score explains so little.
What follows is a method, not a scoreboard. It describes what to measure, how to build a task set, how to score it without kidding yourself, and why the numbers a vendor publishes are usually not comparable with yours. Where we use a figure from our own stack, we label it as ours and say what it covers.
Task success is the only metric with a receipt
Define the task before you define the metric. A task is a caller goal with a verifiable end state: reschedule an appointment, confirm a delivery address, pay an outstanding amount, complete a KYC step, cancel a policy. If the end state cannot be checked in a system of record, it is not a task for benchmarking purposes. It is a conversation, and conversations are hard to score consistently.
Task success is then binary per task, with a partial credit option you define in advance. The agent either completed the goal, completed it with a correction the caller had to make, or failed. The middle case matters because it is where most real deployments sit: the booking exists, but the agent first read back the wrong date and the caller had to repeat it. Counting that as a clean success inflates the number and hides the work that still needs doing.
Figure 2. Task success is checked at the last step. The transcript is evidence of the conversation, not proof that the task completed.
The receipt is the system state, not the transcript. A transcript can read perfectly while the tool call wrote the wrong slot. Check the CRM, the calendar, the order record or the payment ledger after the call. If the benchmark cannot verify the outcome independently of the conversation, it is measuring eloquence.
Turn-taking on numbers, dates and names
The most common failure in Indian-language voice agents is not accent. It is the exchange of information that has no context to help the recogniser: a ten-digit mobile number, a pincode, a date spoken as a festival, a name spelled the way the family spells it. In a code-mixed sentence, the number may be English, the grammar Marathi and the confirmation Hindi. Each of those is a chance to lose a digit.
The fix is conversational rather than acoustic. Ask for the number in groups, read it back in the same grouping, and confirm before the tool call. Do not guess a pincode from a partial match. Do not assume a date means this year. When the caller says something ambiguous, the correct behaviour is a second question, and a benchmark should reward the agent for asking rather than punish it for the extra turn.
Figure 3. The confirmation step that prevents most booking errors. It is a design choice, and it is measurable.
Score turn-taking separately from task success, because the two can diverge. An agent that completes the booking after three confirmations has a high task success rate and a poor efficiency score. An agent that completes it in one turn but with a wrong phone number has the opposite. Report both, and decide which one your business cares about before you look at the results.
Latency: p50, p95 and what is inside the timer
A single latency figure is usually a best case. The useful figures are p50, which describes the typical turn, and p95, which describes the turn that makes a caller say hello again. Report both, report the sample size, and say where the timer starts. The honest definition for a voice agent is the time from the end of the caller’s speech to the first audible response, on a real telephone line, with the same model and voice configuration you would deploy.
Inside that interval there are several components: end-of-speech detection, the final transcription, the model’s first token, any tool call, and the first chunk of synthesised audio. Endpointing alone can add several hundred milliseconds if the silence threshold is conservative, which is why a vendor that measures from the end of the transcript rather than the end of the speech will always look faster. Ask which clock is being used before you compare two numbers.
Figure 4. An illustrative split of a 420 ms turn: 120 + 180 + 90 + 30 = 420. The components are a sample budget, not a measurement of any vendor.
Our published figure is a 420 ms median, p50, for voice in the India region. Read that as one number from one stack under one definition, not as a league table position. For planning, assume the p95 will be materially worse and test it on a mobile network at 6 p.m., not on office wi-fi at 11 a.m. A caller notices a two-second pause far more than a slightly synthetic vowel.
Consent is a scored step, not a footer
If the agent records the call or processes personal data, consent is part of the task. Under the DPDP framework the person needs a notice they can understand and an affirmative action that records agreement, with a path to withdraw. On a voice call that becomes a step in the dialogue: state the purpose, ask, wait for an answer, record the answer with a timestamp and the notice version. It is not a sentence played at the start while the caller waits to speak.
Score three things. Was the notice played in full, in the right language, before any recording that required it? Was the answer captured and stored against the contact? Did the agent stop when the answer was no, and did it offer the alternative channel? An agent that completes a booking but skips the consent step has not passed the task, however pleasant the call sounded.
Figure 5. The consent branch inside a call. A declined answer is a valid outcome and the agent must have a path for it.
This is also where outbound calling needs care. India has rules on unsolicited commercial communication, with consent and do-not-disturb obligations that predate the DPDP Act. The DLT framework most teams know from SMS is not the whole picture for voice. Check the current TRAI requirements and your telephony provider’s policy before you dial a list, and keep the consent evidence next to the number rather than in a marketing spreadsheet.
Escalation correctness, not escalation rate
A low escalation rate looks efficient until you read the calls that should have been escalated. An agent that keeps a frustrated caller in a loop to protect its containment number is worse than one that hands over early. The metric to report is escalation correctness: for each call, was a human needed, and if so, was the handoff timely, warm and complete? False escalations and missed escalations are different failures and should be counted separately.
A warm handoff carries context. The human should see the transcript, the intent, the tool calls already attempted and the reason for escalation, without asking the caller to repeat everything. A cold transfer that drops the caller into a queue is a failure even if the escalation was correctly triggered. Score the handoff itself: did the caller repeat themselves, did the human have the record, was the wait disclosed before the transfer.
Figure 6. Two failures that point in opposite directions. Optimising one without measuring the other moves the problem rather than solving it.
Define the escalation policy before the benchmark, in writing, with the business owner. Which intents must always reach a human? Which amounts, complaints or vulnerable-customer signals trigger a transfer? What happens outside business hours? Without that policy, two raters will disagree about the same call, and the disagreement is about the policy rather than the agent.
Building a task set you can defend
A benchmark task set is a sample, and the sample decides the result. Start from real call recordings or chat transcripts, with identifying details removed, and pick tasks that cover the volume and the difficulty of your actual traffic. A set built entirely from easy, well-formed requests will produce a flattering number that predicts nothing about a Monday morning queue.
Stratify deliberately. Include the high-volume intents, the long tail, the calls that involve money, the calls from older or less tech-comfortable speakers, the code-mixed calls, and the calls with background noise. Include at least a few tasks that should end in escalation, because an agent that never escalates is not being tested on the hardest part of the job. Keep a holdout set that nobody tunes against.
Figure 7. Six strata for a task set drawn from real traffic. The long tail is where the interesting failures live.
Document the provenance. How many calls, from which channel, in which languages, recorded on what kind of line, anonymised how, collected over which weeks. That paragraph is what makes the number defensible. Without it, a benchmark is a demo with a spreadsheet attached, and a reviewer has no way to tell whether it represents anything.
Scoring without kidding yourself
Two raters, one rubric, written before the first listen. Each call is scored on the four dimensions, with a short note explaining any failure. The raters listen independently, then compare. Where they disagree, a third person adjudicates and the rubric is amended if the disagreement reveals an ambiguous rule. That loop is the actual quality work; the score is a by-product of it.
Track agreement. If the two raters agree on almost everything, the rubric is probably too coarse or the sample too easy. If they disagree often, the definitions are unclear and the numbers should not be published until they are fixed. A simple agreement percentage is enough to start; a chance-corrected statistic is better if you want to compare across rounds. What matters is that the reader can see how much judgement went into the score.
Figure 8. The scoring sheet for one call. The notes column is where the usable information lives; the score is downstream of it.
Keep the failures. A benchmark that reports only the aggregate hides the one thing a buyer needs: the shape of the mistakes. Publish a failure taxonomy with counts, for example wrong date confirmed, number misheard, consent skipped, escalation triggered late. The taxonomy is more useful than the score because it tells you what to fix and what to test in your own context.
Why vendor numbers do not compare
Two vendors can both report a word error rate and mean entirely different things. One normalises digits and punctuation before scoring, the other does not. One scores on read speech from a studio, the other on telephone audio. One includes named entities, the other strips them. One reports the mean across a balanced set, the other reports the median of the easy half. None of this is necessarily dishonest; it is simply not a common measurement.
Mean opinion score has the same problem with an extra layer. It is a subjective rating collected from listeners under a defined protocol, and the protocol matters: the number of listeners, their language background, the audio bandwidth, the reference samples and the instructions. A score from a lab test on wideband audio does not transfer to an 8 kHz phone call in a noisy shop. Comparing a vendor’s MOS with a competitor’s is comparing two different experiments.
Figure 9. Two measurements of one call. The left column is a diagnostic; the right column is the outcome the business is paying for.
The same applies to latency. A vendor may measure from the end of the transcript, exclude the tool call, report the median of a fast network, or quote the model’s first token rather than the first audio. Ask for the definition, the percentile, the sample size and the audio conditions. If the answer is not available, treat the number as marketing and run your own test on your own line.
A reporting template that survives scrutiny
A useful report fits on one page and answers five questions: what was tested, on what sample, with what definitions, what happened, and what failed. Put the method before the result. A reader who does not trust the method will not trust the number, and a reader who does trust the method can interpret a modest result correctly.
Report per stratum, not only in aggregate. A 78% overall task success rate could mean 95% on billing and 40% on complaints, and those two facts imply different deployment decisions. Report the confidence interval or at least the sample size per stratum. Report p50 and p95 latency for completed turns, and the share of turns that had to be repeated. Report consent capture as a rate, and escalation correctness as a confusion matrix rather than a single figure.
Figure 10. One page, method first. The failure taxonomy is deliberately part of the result rather than an appendix.
State the limitations in the document, not in a conversation. Which accents were underrepresented, which intents were excluded, how the recordings were cleaned, what the raters disagreed about. A limitations section is not a weakness; it is the difference between a benchmark and an advertisement. The teams that buy voice agents well are the ones that read that section first.
Run your own five-day benchmark
Day one is selection. Pull recordings, anonymise them, write the task list and agree the rubric with the business owner. Day two is setup: configure the agent with the production prompt, voice and tools, and check that the tool calls write to a test system rather than production. Day three is the run, with a person listening live or recording every call. Day four is independent scoring and adjudication. Day five is the write-up and the decision.
The decision should be a threshold agreed in advance. For example: the agent must complete at least four in five tasks in the top two strata without a correction, must never skip consent, and must keep p95 latency below the point where callers start talking over it. If the threshold is missed, the outcome is not a failed vendor; it is a list of fixes and a date to retest. That is a healthier conversation than a demo review.
Figure 11. Five working days, one owner, two raters. The threshold is agreed before the run, not after the numbers arrive.
And if the result is good, resist the urge to generalise. A benchmark on two hundred calls proves something about those two hundred calls. Production will bring different callers, different noise and different edge cases. Keep sampling after launch, score a small batch every month, and treat the benchmark as a habit rather than a certificate.
Frequently asked questions
What is a good word error rate for Indian languages?
There is no defensible universal answer. WER depends on the language, the domain, the audio conditions, the reference transcript and the normalisation rules, and code-mixed speech has no standard reference at all. A number that is good on read Hindi news text can be poor on Marathi billing calls. Use WER as a diagnostic on your own audio, and use task success for the buying decision.
Is a lower latency always better?
Up to a point. Callers dislike long silences, but an agent that replies instantly can feel like it is not listening, and it may talk over a caller who paused to think. What matters is the distribution and the turn-taking behaviour: a p50 that feels natural, a p95 that does not stall, and barge-in that stops the agent when the caller speaks. Measure the whole turn, not one component.
How do you test a voice agent in Hinglish?
Use real code-mixed recordings rather than scripted sentences. Score the task outcome, not the transcript, because a correct booking can come from an imperfect transcript and a perfect transcript can still write the wrong slot. Include numbers, dates and names spoken the way your callers speak them, and count every confirmation turn the agent needed. Publish the failure examples alongside the score.
Why not just use the vendor’s benchmark?
Because it was run on their task set, their audio and their definitions. That does not make it false; it makes it non-comparable. Vendors also tend to publish the aggregate and not the failures, which is the part you need. Ask for the method, the sample, the per-stratum results and the failure taxonomy. If those are not available, run your own test on a small set before you sign.
What does mean opinion score actually measure?
How natural a group of listeners finds a sample of speech, collected under a defined listening protocol. It says nothing about whether the agent understood the caller, completed the task or asked for consent. It is also sensitive to audio bandwidth, listener background and the reference samples used. Treat it as one input to voice selection, not as a quality score for the agent.
How many calls do I need for a useful benchmark?
Enough to cover each stratum with more than a handful of examples, which for most businesses means a few hundred calls rather than a few dozen. The exact number depends on how much the results vary and how confident you need to be. Report the sample size per stratum so a reader can judge. A small, well-documented set is more useful than a large, undocumented one.
What are the consent requirements for an AI voice call in India?
The DPDP framework requires a clear notice and a recorded affirmative consent for processing personal data, and India’s unsolicited commercial communication rules add consent and do-not-disturb obligations for outbound calls. In practice the agent should state the purpose, ask, wait for an answer and log it with a timestamp and notice version. Confirm the current TRAI requirements and your telephony provider’s policy before dialling a list.
Does LoopNow publish its own benchmark results?
We publish a 420 ms median, p50, voice latency for the India region, and we describe our evaluation method on this page. We do not publish a comparative accuracy table, because a table built on our task set would not transfer to yours. If you need evidence for a procurement decision, we can run your task set with you on a sandbox and let you score the calls.
What is the biggest cause of failed voice tasks?
In our experience it is information exchange rather than accent: numbers, dates, pincodes and names, especially in code-mixed sentences. Agents that group digits, read them back and confirm before the tool call fail less often than agents that guess. That is a dialogue design fix, and it shows up in the benchmark as fewer corrections and fewer wrong records rather than as a better transcription score.
