Every vendor blog about "low latency voice AI" is really about one box in the pipeline. A new voice model that shaved 80ms off synthesis. An endpointing tweak. A caching trick. Each is real — and each is useless if the other five stages in your call path are eating 1,200ms while you celebrate the 80.
Latency is a budget, not a feature. A caller hears one number: the silence between when they stop talking and when your agent starts. That number is the sum of every stage from microphone to speaker, plus the network you don't control. This is the end-to-end model — what each stage costs, where the milliseconds actually hide, and where streaming, endpointing, and caching genuinely buy you time versus where they just move the problem.
Why latency is the line between "AI" and "a real conversation"
Human turn-taking has a rhythm. In natural conversation, the gap between speakers averages around 200ms, and people start planning their reply before the other person finishes. When a phone agent leaves 1.5 seconds of dead air after every sentence, the caller doesn't think "impressive AI." They think the line dropped, they talk over the agent, or they hang up.
The practical thresholds we design against:
- Under ~500ms response time feels conversational. The caller barely notices the gap.
- 500–800ms is acceptable but audibly "assistant-like." Fine for many use cases.
- Over ~1,000ms breaks the rhythm. Callers interrupt, repeat themselves, and trust erodes.
That sub-second target is the whole game. And it's not one component's job — it's the budget you allocate across the entire stack.
The latency budget: every stage from mic to speaker
Here's the round trip for a single conversational turn. Rough production ranges for a well-built agent on a phone call:
| Stage | What happens | Typical budget |
|---|---|---|
| Audio ingest / network in | Caller audio reaches your STT (PSTN → SIP → your edge) | 50–150ms |
| STT (streaming) | Speech → partial + final transcript | 100–300ms |
| Endpointing | Deciding the caller actually stopped | 200–800ms |
| LLM TTFT | Prompt → first output token (time to first token) | 200–600ms |
| TTS first byte | First text chunk → first audio out | 80–300ms |
| Audio egress / network out | Synthesized audio back to caller | 50–150ms |
Add those and you're between roughly 700ms and 2.4s. The spread is enormous — and notice what dominates it: endpointing and LLM TTFT, not the STT or TTS models everyone benchmarks.
Two rules fall out of this table immediately:
- Optimize the biggest box first. Swapping a 120ms TTS for a 90ms TTS is noise if your endpointer waits 700ms. Measure before you optimize.
- Stages overlap when you stream. The budget above is the serial worst case. Streaming lets STT, the LLM, and TTS run concurrently instead of end-to-end — which is where most of your real wins come from (below).
Endpointing & silence detection: the most underrated 500ms
Endpointing is the decision: has the caller finished their turn, or just paused to breathe? It's the single largest controllable chunk of latency, and the hardest tradeoff in the stack.
Naive approach: wait for N milliseconds of silence, then fire. Set N too low (say 200ms) and you cut people off mid-sentence — brutal when someone reads a 16-digit card number or says "my address is... 42 Oak Street." Set N too high (900ms) and every turn feels sluggish.
What actually works in production is adaptive, semantic endpointing rather than a fixed silence timer:
- VAD (voice activity detection) gives you the raw "is there audio energy" signal — fast, but dumb about intent.
- Semantic endpointing uses the partial transcript to judge completeness. "My account number is" is obviously unfinished; "I want to cancel my account" is a complete thought. A model that reads the partial text can fire earlier on complete utterances and wait longer mid-number.
- Context-aware timeouts. When you've just asked for a phone number, widen the silence window and don't treat inter-digit pauses as end-of-turn. Tie the endpointing threshold to the slot you're collecting.
Get this right and you reclaim 300–500ms on most turns without ever cutting a caller off. Get it wrong and no model swap will save you.
Streaming everything: the free latency you're leaving on the table
The biggest lie in the serial budget table is that stages happen one after another. They shouldn't. Stream at every boundary and the pipeline collapses:
- Partial transcripts. Don't wait for the final STT result. Feed partials to your endpointer and even pre-warm the LLM context so you're not starting cold when the caller stops.
- Token streaming from the LLM. Time to first token is what matters, not time to full completion. As soon as the first clause streams out, hand it to TTS. You don't need the whole response to start talking.
- TTS chunking. Synthesize and start playing the first sentence while the LLM is still generating the third. First-byte audio, not full-clip audio, is the number that counts.
Done well, the caller hears the first words of the response while the LLM is still writing the rest of it. That's how a stack whose serial budget sums to 1.8s delivers a 600ms felt response. Streaming isn't an optimization you add later — it's the architecture you start from.
One caveat: streaming TTS mid-generation means you're committed to words before the full response exists. If your LLM might self-correct ("actually, let me check that — no, you're right"), you'll have already spoken the wrong half. Constrain the model to commit-once responses, or buffer the first clause until the sentence structure is stable.
Audio caching and pre-warming: real wins, real footguns
Caching is where the Vapi-style "we made it faster" posts live, and it's genuinely useful — with sharp edges.
What's safe to cache or pre-warm:
- Fixed prompts and boilerplate. Greetings, hold messages, disclosures, "please hold while I look that up." Pre-synthesize these once. Zero TTS latency at runtime.
- Connection warm-up. Keep STT/LLM/TTS sockets open and models warm so the first turn of a call doesn't pay cold-start tax. This alone can save hundreds of milliseconds on turn one.
- Predictable filler. A short, natural "let me check that for you" played while a slow tool call runs masks backend latency the caller would otherwise hear as dead air.
What breaks if you cache it:
- Anything with dynamic data. Cache "Your balance is [X]" as a template and you'll eventually read the wrong balance to the wrong caller. Cache the carrier phrase, synthesize the variable slot live.
- Personalized or compliance-sensitive audio. Names, account details, quoted prices — synthesize fresh, every time.
- Stale voice/model versions. A cached clip from an old TTS voice next to a live one is jarringly obvious mid-sentence. Version your cache keys to the voice model.
Caching buys you the most on the predictable parts of a call. It does nothing for the dynamic reasoning turn — which is exactly why it can't be your whole latency story.
Telephony & network reality: the last mile you don't own
You can tune every model to perfection and still ship a laggy agent, because a phone call rides infrastructure you don't control.
- PSTN and SIP add real time. The public switched telephone network and SIP trunking introduce transport delay before your STT ever sees audio. Codec transcoding (e.g., to/from G.711) adds a little more.
- Jitter and packet loss. Wireless callers, bad Wi-Fi, and congested carriers deliver audio unevenly. Your jitter buffer smooths playback but adds latency to do it — another tradeoff to tune, not eliminate.
- Geography. If your inference runs in us-east and your caller and telephony provider are in Europe, you've added a transatlantic round trip to every turn. Co-locate your media path with your callers and your model endpoints.
- The media path matters as much as the model. WebRTC-to-SIP bridging, SBC routing, and where your media server sits can cost or save more than a model swap. This is infrastructure engineering, not prompt engineering.
The takeaway: budget 100–300ms for network and telephony you can't optimize away, and make sure the milliseconds you can control aren't being wasted on a media path that hairpins across three regions.
Measuring turn-taking latency in production (the number that matters)
You cannot optimize what you don't measure, and the metric that matters isn't any single component's benchmark. It's end-of-user-speech to start-of-agent-audio — measured in production, on real calls, at the percentile that hurts.
- Instrument the full round trip. Timestamp: last caller audio frame → endpoint fired → LLM first token → TTS first byte → first audio frame out. Log every stage on every turn so you can see which box blew the budget when a call felt slow.
- Watch p95, not the average. Your mean latency can be a lovely 550ms while p95 is 1.8s — and it's the p95 turns that make callers hang up. Averages hide the calls you're losing.
- Track barge-in / interruption rate. If callers frequently talk over your agent, your endpointer is firing late or your response is starting slow. It's a latency symptom disguised as a UX metric.
Ship the instrumentation before you tune. Every "we cut latency" claim that isn't backed by production p95 measurement is a guess.
Internal links
- How We Built the Voice AI Pipeline — STT choices: Whisper vs Deepgram
- Fixing SIP Latency in AI Call Center Solutions
- Bridging WebRTC and SIP: Low-Latency AI Voice Agent Handoffs
- AI Voice Agents for Enterprise: Architecture & ROI
- How to Manage High Call Volumes Without Hiring More Agents (2026)
FAQ
What is a good latency target for a production voice AI agent? Aim for under 800ms end-to-end (end of caller speech to start of agent audio), and treat sub-500ms as the goal for a genuinely conversational feel. Measure at p95, not average — the slow tail is what drives hang-ups.
Which part of the voice AI pipeline causes the most latency? Usually endpointing and LLM time-to-first-token, not the STT or TTS models most benchmarks focus on. Endpointing can add 200–800ms on its own, so it's the first place to look before swapping any model.
Does audio caching reduce voice AI latency? Yes, for fixed content — greetings, disclosures, hold messages, and filler phrases can be pre-synthesized for near-zero runtime latency. It does nothing for dynamic reasoning turns, and caching anything with personalized or variable data risks reading the wrong information to a caller.
Why does my voice agent feel laggy even with a fast model? Because latency is the sum of the whole pipeline plus telephony you don't control. A fast TTS won't help if your endpointer waits 700ms, your LLM streams slowly, or your media path hairpins across regions. Instrument every stage and fix the biggest box first.
CTA
Finn is built latency-first — streaming STT, semantic endpointing, and token-to-TTS handoff tuned to hit sub-second turn-taking on real phone calls, not benchmark clips. See how Finn's voice agents handle production call volume without the dead air.
