VoIP latency is the delay between someone speaking and the other party hearing it. ITU-T Recommendation G.114 puts the ceiling for normal conversation at 150 milliseconds one-way — beyond roughly that, people start talking over each other.
For a voice AI agent the budget is tighter still, because the agent has to think inside it.
Where the milliseconds go
On a human call, latency is transport: packetisation, network transit, jitter buffer, playout. A well-run VoIP path sits comfortably inside G.114.
A voice agent adds a processing stack between the two humans' worth of transport:
- Capture and encode the caller's audio
- Speech to text, which cannot finish until the caller stops speaking
- Endpointing — deciding they have stopped, which is a guess and a cost
- The model generating a response
- Text to speech producing audio
- Transport back to the caller
Each step is small. The sum is not. And the parts do not add up the way an architecture diagram suggests, because one of them dominates in a way that is easy to miss.
Time to first audio is the number
Total processing time is the wrong metric. What the caller experiences is the gap before they hear anything.
An agent that takes 900ms to produce a complete response but starts speaking at 300ms feels responsive. One that takes 600ms total but says nothing until 600ms feels slow. Streaming changes the perceived latency far more than raw speed does, which is why chasing a faster model often disappoints while chunking the response helps immediately.
This is also why endpointing sits on the critical path. Every millisecond spent deciding the caller has finished is a millisecond before anything else can start. Aggressive endpointing cuts latency and interrupts people; conservative endpointing is polite and slow.
The budget in practice
Working backwards from a conversational feel:
- Under ~800ms mouth-to-ear reads as natural
- 800ms–1.2s is noticeable but tolerable
- Beyond ~1.2s callers begin talking over the agent, and every collision costs a repair turn
Against that, transport alone on a domestic call might take 80–150ms round trip. International adds more, and it is not optional — physics sets a floor. A pipeline routing audio to a distant region spends a large share of the budget before any processing starts, which is the usual reason a demo that felt instant in testing feels sluggish in production.
The voice latency calculator adds the stages up so you can see which one is actually spending your budget, rather than assuming it is the model.
Measuring it honestly
Measure mouth-to-ear on a real call over a real carrier. Model inference time in isolation is not latency; it is one term in the sum, and usually not the largest.
Measure the tail, not the mean. A pipeline averaging 700ms with a p95 of 2 seconds is not a 700ms pipeline — one call in twenty is unusable, and those are the calls people remember.
See also voice latency for the individual terms.
Frequently asked questions
What is an acceptable latency for VoIP? ITU-T G.114 recommends keeping one-way delay under 150ms for normal conversation. Voice agents need a tighter budget because processing sits inside the same window.
Why does my voice agent feel slow when the model is fast? Usually endpointing and time-to-first-audio rather than inference. If the agent waits to finish generating before speaking, the caller hears the whole pipeline instead of the start of a sentence.
Does jitter matter as much as latency? It matters differently. Jitter is absorbed by a buffer, and that buffer itself adds delay — so reducing jitter can reduce latency indirectly.
Is 500ms good enough? For mouth-to-ear, usually yes. Under about 800ms reads as conversational; problems start past roughly 1.2 seconds.
