p50 / p95 latency
Median and tail response time. Voice quality hinges on p95 rather than a best-case demo, so always state the percentile and region.
p50 is the median; p95 is the slow tail. Voice quality depends on p95, not a single best-case demo. Always state the percentile and region.
p50 is the median; p95 is the slow tail. Voice quality depends on p95, not a single best-case demo. Always state the percentile and region.
All glossary terms
The short version
- p50 is the median: half of the measured turns were faster, half were slower. p95 is the value below which ninety-five percent of turns fall, so one turn in twenty is worse.
- Averages mislead because latency distributions have a long right tail. A mean of 900 milliseconds can hide a p95 of four seconds, and the four-second turns are the ones callers remember.
- In voice, the number that matters is total turn latency: the gap from the moment the caller stops speaking to the moment they hear the first syllable of the reply.
- Model latency is one component, usually time to first token. It is not the turn. Endpointing, speech recognition, network round trips and speech synthesis each add their own delay.
- Percentiles cannot be averaged across windows. Five days of p95 values do not produce a weekly p95. Merge histograms, or store every sample.
- State the percentile, the sample size, the region and the measurement point. A p95 without those four facts is a marketing number.
What a percentile is, and what p95 is not
A percentile is a position in a sorted list. If you record a thousand turn latencies and sort them from fastest to slowest, the p50 is the value at position five hundred, and the p95 is the value at position nine hundred and fifty. Formally, using the nearest-rank method, p95 is the observation at the ceiling of 0.95 times the sample count. Half of your callers experience better than the median and half experience worse. Ninety-five percent experience better than p95 and five percent experience worse.
That last sentence is the whole point and it is routinely dropped from vendor pages. A p95 of 900 milliseconds does not mean a caller waits 900 milliseconds. It means one turn in twenty waits longer, possibly much longer. Human perception does not average either. A call with fifteen snappy replies and one four-second silence is remembered as a slow call, because the silence is what the person noticed.
What p95 is not: it is not a maximum, and it is not a guarantee. The worst turn may be eight seconds even when p95 is under one. It is also not a property of your system in general. It is a property of a sample, measured at a point, in a region, on a carrier, during a period. Change any of those and the number changes. A p95 quoted without them is a temperature reading with no thermometer.
Why the average is the wrong summary
An average is a sum divided by a count, and it is pulled by whichever values are largest. When most turns take 700 milliseconds and a few take six seconds because of a retry, the mean lands somewhere around 900 and describes nobody. No caller experienced 900 milliseconds. They experienced 700 or they experienced 6,000, and the mean describes a synthetic person who does not exist.
Latency distributions are also frequently bimodal. A voice agent that hits a cache takes 400 milliseconds; the same agent on a cache miss calls the model and takes 1,900. The average sits at 1,150, in the valley between the two modes, where almost no turn lands. Reporting the mean of a bimodal distribution is not a simplification, it is a category error. You need the two modes and their proportions, which is what a histogram gives you and a mean destroys.
Percentiles have their own limits. Two systems can share a p95 and behave completely differently: one is consistently slow for a fifth of calls, the other is fast for everyone except a pathological input that hangs. Report p50, p95 and the maximum together, and keep a histogram. The three numbers together describe the shape well enough to make a decision, and the shape is what determines whether callers stay.
You cannot average percentiles across days
This is the most common analytical error in latency reporting, and it survives code review because it looks reasonable. A dashboard computes p95 for each hour, then the weekly report takes the arithmetic mean of those twenty-four values. The result is not the weekly p95. It is the average of hourly p95 values, which is a different number and usually a flattering one, because a quiet hour with three calls contributes as much as a busy hour with three thousand.
The correct method is to keep the raw samples or a mergeable sketch. HDR histograms and t-digest structures merge exactly or approximately, and a merged weekly histogram yields a true weekly percentile. If you cannot keep samples, keep per-bucket counts at a fixed set of latency boundaries and sum the counts before computing the percentile. Both approaches cost little and both are far cheaper than the argument that follows a number nobody can reproduce.
There is a second, subtler version of the same error: averaging across regions or carriers. A p95 measured in Mumbai on one carrier and a p95 measured in Delhi on another are two different populations. Combining them into one national number hides the region that is failing. Segment first, then aggregate only if the segments are comparable, and label the aggregate with what went into it.
How many samples does a p95 need
A p95 estimated from twenty calls is not a p95. It is the second slowest of twenty, and the next twenty calls will produce a different second slowest by a wide margin. The reason is arithmetic: if five percent of turns are above p95, then a sample of twenty contains about one such turn, so the tail estimate rests on a single observation. A sample of two hundred contains about ten, which is still noisy but usable. A sample of two thousand contains about a hundred, which is enough to compare two weeks.
This has a direct operational consequence. If you want to monitor p95 per hour per region per carrier, and you only take four hundred calls a day, you do not have the volume to support that granularity. You will get a jagged line that moves because of sampling, and someone will raise an incident because a quiet hour had three bad calls. Either widen the window, drop a dimension, or report the count alongside the percentile so that readers can see how thin the evidence is.
For tail metrics such as p99, the requirement multiplies. Estimating p99 from a thousand calls gives you ten observations above the line, which is barely enough to say anything. Most teams that quote p99 are quoting a handful of worst cases with a statistical label attached. If what you actually care about is the worst experience, report the maximum and the count of turns above a threshold, which are honest and easy to verify.
Where a voice turn actually spends its milliseconds
A voice turn is not one latency. It is a chain, and the caller experiences only the sum. The chain begins with endpointing: the system has to decide the caller has stopped speaking, and that decision costs time. A conservative voice activity detector waits for silence and adds three to seven hundred milliseconds. An aggressive one cuts people off mid-sentence. Every voice product is trading one of those failures against the other, and the p95 is dominated by whichever choice was made.
Then the final speech recognition result has to stabilise, which in a streaming system is usually fast but not free. The text goes to a model, and the first token comes back after a time-to-first-token that depends on prompt length, model size, queue depth and region. The reply is then synthesised, and the caller hears nothing until the first audio chunk is ready. Network and telephony add a round trip on each direction, plus jitter buffer and codec delay on the carrier side.
A planning budget for a conversational agent, stated as a target rather than a measurement, might allocate 300 milliseconds to endpointing, 150 to recognition finalisation, 450 to model first token, 250 to first audio, and 120 to network, for roughly 1.27 seconds. That is a budget a team can argue about component by component. It is much more useful than a single number from a demo, because when the p95 slips you can see which term moved.
Model latency is not turn latency
Model providers publish latency figures that describe the model in isolation: time to first token, tokens per second, sometimes an end-to-end number for a short prompt. Those numbers are real and they are not the caller’s experience. A model with a 300 millisecond first token inside a turn that also spends 600 milliseconds waiting for silence and 400 milliseconds on a carrier round trip produces a 1.3 second gap. The model is fast. The turn is slow.
The relevant model metric for voice is not time to last token, because the caller does not wait for the full answer. It is time to the first speakable clause, which is why streaming and sentence chunking matter more than raw throughput. A model that emits two hundred tokens per second but only after buffering a whole paragraph will feel slower than one that emits sixty tokens per second and streams a clause at a time. Throughput is a batch metric being applied to an interactive problem.
The other model-side variable is variance. A provider with a p50 of 400 milliseconds and a p95 of 2.4 seconds will produce noticeably worse calls than one with a p50 of 600 and a p95 of 900, even though the median is better. For voice, the tail is the product. When you evaluate a provider, ask for the p95 at your prompt length in your region, and if they cannot supply it, measure it yourself with your own traffic rather than a benchmark prompt.
The network and the carrier are part of your latency
Indian voice traffic crosses PSTN interconnects, mobile networks and, often, a SIP trunk before it reaches your media server. Each hop contributes. ITU-T G.114 recommends keeping one-way delay for voice under about 150 milliseconds for most users, with degradation becoming noticeable beyond 400. That recommendation is about conversational quality, and it does not include your application processing. It is the floor beneath your budget, not a target you can spend.
Mobile access adds jitter, which is variation in packet arrival time, and jitter buffers convert that variation into delay. A caller on a congested cell in a dense market may see a buffer of a hundred milliseconds on a good minute and several hundred on a bad one. Packet loss makes it worse, because retransmission and concealment both cost time. RFC 3550 defines the RTP measurements that let you see this, and most media stacks expose jitter and loss per call if you ask.
The practical implication is that a latency figure measured on a laptop over office Wi-Fi is not a measurement of the product. It is a measurement of the office. Test on a real handset, on at least two carriers, in at least two cities, at a busy hour. The gap between a fibre-connected demo and a 4G call at 7pm is routinely several hundred milliseconds, and it is entirely outside your code.
Barge-in makes the tail harder, not just faster
Barge-in is the ability for a caller to interrupt the agent mid-sentence. It changes the latency problem in two ways. The detection path has to run while the agent is speaking, which means echo cancellation has to separate the agent’s own voice from the caller’s, and that separation is imperfect on speakerphone and in noisy rooms. And the stop path has to be fast: the agent must cease audio within a couple of hundred milliseconds or the caller hears the agent talking over them, which reads as broken rather than slow.
The measurement is therefore different from turn latency. Barge-in latency is the interval from the caller’s first syllable to the last audio frame the agent emits. It is not the interval to the next reply; that comes later. A system can have excellent turn latency and terrible barge-in, because the interruption path shares nothing with the reply path except the microphone. Instrument both, and record them separately.
There is a design trade as well. A very sensitive barge-in detector will trigger on a cough, a background conversation or a television, and the agent will stop mid-sentence for no reason. A very conservative one will let the caller talk over a full sentence. The setting is a product decision about which failure is worse in your context, and the right value for a collections call differs from the right value for a restaurant booking.
Measuring end to end without fooling yourself
Instrument at the edges, not in the middle. The start marker is the last frame of caller speech, captured by the media server, and the end marker is the first frame of agent audio leaving the media server. Everything between those two points is the turn. If you start the clock when the model request is sent, you have measured the model and excluded the parts that usually regress.
Use one clock. Distributed systems with unsynchronised clocks produce negative durations and heroic explanations. Timestamp at the media server with a monotonic clock, or use a trace identifier propagated through every hop with a single time source. Then store the raw duration with dimensions: region, carrier, model, prompt version, endpointing configuration, and whether the turn hit a cache. Without dimensions you can compute a percentile and cannot explain it.
Test with both synthetic and real traffic. Synthetic calls give you a controlled script, a repeatable prompt and a clean baseline, and they are indispensable for regression testing after a deploy. Real calls give you the distribution that customers actually experience, including accents, background noise and Hinglish code-switching that no script contains. A synthetic suite that reports 700 milliseconds while production p95 is 2.1 seconds is not lying; it is measuring a different population.
Setting an SLO you can defend
A service level objective for voice is a target for a proportion of turns, not for all of them. A defensible shape is: ninety-five percent of turns complete within 1.5 seconds, measured at the media server, across real traffic, in the region where the call terminated, over a rolling seven days. That sentence contains the percentile, the threshold, the measurement point, the population and the window. Each of those is a decision, and any of them can be argued about, which is the sign of a real objective.
Pair it with an error budget. If the objective is ninety-five percent, five percent of turns may exceed the threshold before the service is out of budget. That allows a team to ship a change that costs a little tail latency without a panic, and it forces a conversation when the budget is exhausted. The alternative, treating every slow turn as an incident, produces a team that stops deploying and a threshold that quietly gets relaxed until it means nothing.
Do not set the objective from a competitor’s marketing page. Vendor latency numbers are measured on a benchmark prompt, in one region, at one time, and they usually report p50. Set the objective from your own calls, your own carriers and your own tolerance, then improve it. A 1.5 second p95 that you can prove is worth more than a 400 millisecond claim you cannot reproduce.
Worked example: five hundred calls on a Growth plan
Take a support line handling 500 calls a month at an average of three minutes, which is 1,500 voice minutes and sits exactly at the Growth allowance of 1,500 minutes for ₹7,999 plus 18% GST, or ₹9,438.82. Suppose the agent takes twelve turns per call. A turn that runs 1.2 seconds at p50 but 3.4 seconds at p95 means that in a twelve-turn call, the probability of at least one turn landing in the slow five percent is roughly forty-six percent. Half of all calls contain a noticeable pause.
Now improve the tail rather than the median. Caching the three most common answers, trimming the system prompt and moving the model endpoint to the same region as the media server might leave the p50 almost unchanged while pulling the p95 from 3.4 to 1.6 seconds. The caller does not experience a faster average. They experience fewer silences, and the arithmetic above says the share of calls containing a bad pause falls from about half to about one in five.
That is also where the money is. Calls that contain a long silence are the ones where the caller says are you there, repeats themselves, or hangs up. Reducing the tail reduces repetition, which shortens calls, which reduces minutes consumed against the allowance. The saving is real but secondary. The primary effect is that the agent stops sounding broken, and no amount of median improvement buys that.
Frequently asked questions
What is the difference between p50 and the average?
The average is a sum divided by a count, so one very slow turn drags it upward. The median, which is p50, is the middle value of the sorted list, so half the turns are faster and half are slower. On a skewed distribution they diverge, and the median is usually closer to what a typical caller experiences. Neither tells you about the tail.
Why is p95 more useful than p50 for voice?
Because a conversation contains many turns, and one slow turn is enough to make the whole call feel broken. At twelve turns per call, a five percent per-turn tail means roughly half of calls contain at least one pause. The median describes the best half of the experience and hides the pauses that people actually remember, repeat themselves over, or hang up on.
Can I average daily p95 values to get a weekly p95?
No. The average of percentiles is not a percentile of the population, and it weights a quiet day the same as a busy one. Keep raw samples or a mergeable histogram, merge first, then compute the percentile once. If you cannot keep samples, store counts at fixed latency boundaries and sum the counts before computing.
How many samples do I need before quoting a p95?
A few hundred at minimum, and more if you want to compare periods. With twenty samples, p95 is effectively the second slowest observation, and it will move wildly between samples. Report the sample count next to every percentile, and if the count is small, show the distribution or the maximum instead of pretending to a precision you do not have.
Does model latency determine call quality?
Only partly. Model time to first token is one term in a sum that also includes endpointing, recognition finalisation, speech synthesis and carrier round trips. A fast model inside a slow chain still produces slow calls. Measure the whole turn, and treat the model figure as a component you can optimise once the rest is understood.
What is a reasonable p95 target for a voice agent?
There is no universal number, but many teams target somewhere between 1.2 and 1.8 seconds for the gap between the caller stopping and the agent starting, measured at the media server on real traffic. Anything under a second feels responsive; beyond two seconds callers begin to check whether the line is alive. Set your own target from your own calls.
Why does my latency look worse on mobile calls?
Mobile access adds jitter, packet loss and transcoding, and jitter buffers convert variation into delay. A congested cell at a busy hour can add several hundred milliseconds before your application sees any audio. Test on real handsets, on more than one carrier, at peak times, and segment your dashboards by carrier so the effect is visible rather than averaged away.
Should I measure barge-in separately?
Yes. Barge-in latency is the time from the caller’s first syllable to the last audio frame the agent emits, which is a different interval from the time to the next reply. A system can have good turn latency and poor barge-in, because the stop path and the reply path are different code. Record both, and set the detector sensitivity as a product decision.
What does p99 tell me that p95 does not?
It describes the worst one percent, which is where hangs, retries and provider incidents live. It also requires far more samples to estimate honestly. With a thousand calls, p99 rests on ten observations. If what you care about is the worst experience, report the maximum and the count of turns above a threshold, which are exact and easy to verify.