Barge-in
Letting a caller interrupt a voice agent mid-sentence; the agent has to stop speaking, listen and resume without losing the thread.
Barge-in lets a caller interrupt the agent mid-sentence. Good voice agents stop speaking, listen and resume without losing context.
Barge-in lets a caller interrupt the agent mid-sentence. Good voice agents stop speaking, listen and resume without losing context.
Related
The short version
- Barge-in is the ability of a caller to interrupt a voice agent while it is speaking, and the agent’s ability to stop, listen and continue without losing the thread.
- It is not voice activity detection and it is not endpointing. Detection is one input; barge-in is the whole behaviour, including stopping playback, finalising what was heard and deciding what to do next.
- The engineering problem is a latency budget in milliseconds, not a feature checkbox. Detection, cancellation and acknowledgement each consume part of it.
- False triggers are the hard part. Backchannels such as haan and ok, coughs, television audio and the agent’s own echo all look like interruptions to a naive detector.
- Hinglish calls make it harder, because a two-syllable acknowledgement can change the meaning of a turn and code-switching happens mid-sentence.
- Measure detection latency at p50 and p95, plus false-positive and false-negative rates. An agent that stops at every noise is worse than one that never stops.
Barge-in, defined by behaviour rather than detection
Barge-in is what happens when a person speaks while the agent is still talking. In a good implementation the agent stops within a fraction of a second, keeps the audio it has already heard, finishes recognising the interruption, and responds to it while retaining everything said before. In a bad implementation the agent keeps talking over the caller, or stops and then restarts the same sentence from the beginning, or loses the earlier context entirely.
The term comes from broadcast and studio intercoms, where it meant cutting into a live feed. In voice agents it names a behaviour, not a component. That distinction matters because teams often ship a voice activity detector, see the waveform dip, and conclude that barge-in is done. Detection is necessary and nowhere near sufficient. The caller experiences the stop, the silence, the acknowledgement and the recovery as one event.
What barge-in is not: it is not endpointing, which decides when a speaker has finished a turn. It is not duplex audio, which is a transport property. It is not the same as a hangup, which ends the call. And it is not a per-word interrupt setting on a text-to-speech engine, although that setting is one of the levers. The behaviour spans transport, detection, recognition, generation and playback. Nor does it apply to every agent: an email or WhatsApp agent has no playback to interrupt, so the concept belongs to the voice adapter rather than to agentic AI in general.
The audio path, and where the milliseconds go
A voice call is a chain of conversions. The caller’s speech is captured by a microphone, compressed by a codec, sent over a mobile network, decoded at the media server, segmented into frames, passed to a speech recogniser, turned into text, sent to a model, turned back into audio by a synthesiser, and streamed to the caller’s ear. Barge-in has to intervene somewhere in that chain, and where it intervenes determines how quickly the caller hears silence.
The cheapest intervention is at playback: stop sending audio frames to the caller. That requires the playback buffer to be small enough that stopping takes tens of milliseconds rather than hundreds, and it requires the media pipeline to support cancellation without tearing down the call. The more expensive intervention is at recognition: decide whether the sound was speech, what words it contained, and whether it was directed at the agent. Both have to happen, but they can happen in parallel.
The mistake is to serialise them. If the agent waits for a final transcript before it stops talking, the caller hears the agent continue for another second, which feels like being ignored. If it stops on any energy and waits for a transcript that never contains words, it stops for a cough. The correct design stops early on a cheap signal and decides later on an expensive one, with a way to resume if the decision is that nothing was said.
A latency budget, stated as a target
Conversational turn-taking is fast. People leave gaps of a few hundred milliseconds between turns, and a system that takes a second to yield feels slow even when every individual step is fast. A workable budget for a telephony agent is roughly this: one hundred to two hundred milliseconds to decide that the sound is likely speech, under one hundred milliseconds to stop playback, two to four hundred milliseconds to get a usable partial transcript, and three to six hundred milliseconds for the first token of a new response.
Those are design targets rather than measured results, and they should be treated as an allocation to defend. If the detector is generous, the false-positive rate rises. If the recogniser is asked for a final transcript before stopping, the stop is late. If the model is asked for a full paragraph before synthesis begins, the caller waits again. Each choice spends from the same budget.
In practice the budget is set by the worst network you support, not the best. A caller on a congested mobile link in a Tier 2 city adds jitter and packet loss that no amount of server tuning removes. Design for the p95 caller and accept that the median experience will feel better than the number suggests.
False triggers, and why they are the real problem
A detector that fires on energy alone will fire constantly. Callers say haan, accha, hmm and theek hai while the agent is talking, and those are not interruptions; they are the sounds of listening. A cough, a car horn, a television in the background, a colleague’s voice in the same room and the agent’s own audio leaking back through the caller’s speaker all produce energy that looks like speech.
The practical mitigations stack. Require a minimum duration of speech, not a single frame. Use echo cancellation so the agent does not interrupt itself. Check whether the detected speech contains at least one plausible word within a short window, using the partial transcript rather than waiting for a final one. Suppress a stop when the only recognised tokens are in a backchannel lexicon for the active language. And cap the number of stop-and-resume cycles per turn so that a noisy line cannot produce a stutter loop.
There is a real trade-off here and it should be a product decision, not an engineering accident. Stopping too eagerly produces an agent that cannot finish a sentence. Stopping too slowly produces one that talks over people. For a collections call the second is worse; for a long informational read-out the first is worse. The threshold should be tuned per use case and recorded in the configuration.
Telephony makes it harder than a browser demo
A browser demo has wideband audio, a known microphone and no meaningful packet loss. A phone call has narrowband audio at eight kilohertz, a codec that discards the frequencies that distinguish s from f, a network that reorders and drops packets, and a caller who may be on speakerphone in a moving vehicle. Speech recognition accuracy drops, and the detector’s job gets harder at the same time.
Echo is the specific hazard. When the caller is on speakerphone, the agent’s own voice returns through the microphone delayed and attenuated. Without echo cancellation the detector hears the agent as an interruption and stops it, then resumes, then stops again. The fix is acoustic echo cancellation plus a guard that ignores input correlated with the audio currently being played, and it needs testing on real handsets rather than a softphone on a laptop.
The media path also decides whether stopping is even possible. If the agent streams a whole pre-synthesised file, the server can stop sending packets but the handset may have buffered several seconds. Streaming synthesis in small chunks, with a short buffer, is what makes the stop audible. A vendor that plays a finished audio file and calls the result barge-in is describing something the caller will not experience as interruption.
Hinglish, short acknowledgements and names
Indian callers mix languages inside a sentence, and the interruption is often the shortest part of the call. A caller who says arre ek minute, mujhe amount batado has switched twice in eight words. A recogniser configured for a single language will produce a partial transcript that looks like noise, and the semantic gate described above will then suppress a genuine interruption.
The backchannel lexicon has to be language-aware for the same reason. Haan, ji, accha, theek hai, okay and hmm are all listening sounds, but the same syllables can be a real answer to a yes or no question. The difference is context: is the agent asking a question or making a statement? An agent that treats every haan as an interruption cannot deliver a long explanation; one that treats every haan as assent will record a confirmation the caller never gave.
Numbers, names and amounts are the weakest link. A caller interrupting to correct an amount expects the agent to hear the correction the first time. Reading back the amount, the date or the last four digits of a reference is not a courtesy; it is the error correction mechanism for a channel that mishears. Two failed read-backs should end the automated attempt, not start a third.
Resuming without losing the thread
Stopping is the easy half. The caller interrupted for a reason, and the agent has to answer the interruption and then decide what to do with the sentence it was in the middle of. Three behaviours are defensible. Abandon the sentence and answer the question. Finish the thought in one short clause and then answer. Or answer and explicitly offer to continue, as in the price is confirmed, shall I finish the timeline.
What is not defensible is restarting from the top. A caller who has heard the first two reasons does not want them again. That requires the playback position and the generated content to be tracked at sentence granularity, so the agent knows what was actually heard rather than what was generated. The distinction matters when the interruption arrives during the third sentence of a five-sentence answer.
The conversation state has to survive too. If the caller interrupts with a question about a different account, the agent must answer it and then return to the original task with the original identifiers, not re-ask for verification. This is the point where barge-in stops being a voice feature and becomes a runtime feature: the envelope, the open commitments and the task state live above the audio layer.
Measuring barge-in without inventing numbers
Four measurements describe the behaviour. Detection latency: time from the end of the caller’s first syllable to playback stopping, reported at p50 and p95 over real calls. False-positive rate: interruptions detected where the caller said nothing directed at the agent, sampled and labelled by a human. False-negative rate: cases where the caller clearly tried to interrupt and the agent kept talking, again from a labelled sample. Recovery rate: interruptions after which the task still completed without the caller repeating information.
Every one of these needs a denominator and a date. A detection latency quoted without a percentile, a region and a handset mix is decoration. A false-positive rate quoted without a labelled sample is a guess. The honest report says how many calls were reviewed, by whom, on which dates, and what counted as an interruption.
The qualitative check is still the best one. Listen to twenty interrupted calls end to end and ask whether the agent sounded like it was listening. No dashboard captures the difference between an agent that stopped and one that yielded.
Common implementation mistakes
The first is treating detection as delivery. A waveform that dips in a dashboard is not an experience. The second is a playback buffer measured in seconds, which makes the stop inaudible even when the server reacts instantly. The third is discarding the partial transcript when the interruption is classified as noise, which throws away the evidence needed to tune the classifier.
The fourth is a global setting. Barge-in thresholds that work for a short confirmation flow will fail in a long explanation, and the reverse. Configuration should be per agent and per step, with a record of what was chosen. The fifth is ignoring the recording and consent path: an interrupted call is still a recorded call, and the notice has to have been given before the first word, not after the first interruption.
The sixth is testing only with clean audio. A test set should include speakerphone, traffic, a television, a second speaker in the room and a caller who switches language mid-sentence. Those are ordinary Indian conditions, and they are the ones that decide whether the feature works.
Worked example: the EMI question in the middle
An agent is explaining a restructuring option to a borrower: three revised instalments, a revised date, and a note that the change carries a fee. Forty seconds in, the caller says arre ruko, kitna baaki hai. A naive system keeps talking for another second, then stops and asks the caller to repeat. The caller repeats, irritated, and the agent answers with the outstanding amount but has lost the rest of the explanation.
A working system stops within a couple of hundred milliseconds, classifies the partial transcript as a question rather than a backchannel, answers the outstanding amount with a read-back, and then says the fee is five hundred rupees, shall I finish the two remaining points. The caller says haan, and the agent continues from the third point rather than the first. The whole exchange adds four seconds and removes a repetition.
The design decisions that made it work are not exotic. Small audio frames, echo cancellation, a Hinglish partial transcript, a lexicon that knows ruko is not a listening sound, and a saved position at sentence granularity. None of them is a model improvement, which is why voice quality is usually an engineering problem wearing a machine learning hat.
Frequently asked questions
Is barge-in the same as voice activity detection?
No. Voice activity detection is one input that says speech may be present. Barge-in is the end-to-end behaviour: stop playback quickly, decide whether the sound was a real interruption, recognise it, respond and resume the earlier task. A system with detection and no recovery has not implemented barge-in.
How fast should the agent stop talking?
Fast enough that the caller does not notice a delay. A design target is under one hundred milliseconds from the decision to stop until audio ceases, with detection adding a hundred to two hundred milliseconds. The audible result depends more on the playback buffer than on the server, so test on real handsets over a mobile network.
What counts as a false trigger?
Any stop where the caller did not intend to interrupt: a cough, traffic, a television, a colleague in the room, or the agent’s own echo returning through a speakerphone. Backchannels such as haan and accha are a special case, because they are speech but not interruptions, and they need a lexicon rather than an energy threshold.
Does barge-in work in Hindi and Hinglish?
It can, with two additions. The partial transcript needs to handle code-switching, because a caller may switch language inside the interruption. And the backchannel lexicon needs to be per language, because a word that means yes in one context is a listening sound in another. Without both, the classifier suppresses real interruptions.
Why does the agent interrupt itself on speakerphone?
That is echo. The agent’s own voice leaves the phone speaker, re-enters the microphone delayed and attenuated, and looks like speech. Acoustic echo cancellation plus a guard that ignores input correlated with the audio being played fixes most cases, but it has to be tested on the handsets your callers actually use.
Should the agent restart the sentence after an interruption?
No. Restarting makes the caller hear the same information twice and signals that nothing was retained. Save the position at sentence granularity, answer the interruption, then offer to continue from where you stopped. The exception is a very short sentence, where finishing it first is less awkward.
How do you measure it?
Report detection latency at p50 and p95, false-positive and false-negative rates from a labelled sample, and recovery rate for interrupted calls. State the number of calls reviewed and the dates. Then listen to twenty calls end to end, because no metric captures whether the agent sounded like it was listening.
Does barge-in affect call recording and consent?
The recording continues through the interruption, and the consent notice must have been given before the call, not after. The interruption itself becomes part of the record, including the partial utterance, which is why retention and access rules for call audio matter as much as for the transcript.
Does this need a special telephony provider?
You need a media path that lets you stream audio in small frames and cancel playback without dropping the call. Many providers support that, but not all expose it, and some buffer several seconds at the handset. Ask for the buffer size and the cancellation behaviour in writing before committing to a number.