Whisper vs Deepgram 2025: STT for Voice Agents

Every "Whisper vs Deepgram" post you'll read is written by someone selling one of them. Deepgram's own comparison leads with "90%+ accuracy in 300ms." True, and beside the point. If you're wiring speech-to-text into a real-time voice agent that answers phone calls, accuracy on a clean podcast clip is not the number that decides whether your agent feels human or feels broken.

We build contact-center voice agents at Finn. We've shipped both engines to production. Here's the vendor-neutral version — the one grounded in 8kHz telephony audio, a real latency budget, and the ops bill nobody prices in.

The Real Decision: Streaming STT Is a Latency Budget, Not an Accuracy Leaderboard

A voice agent has one hard constraint: the human on the phone expects a reply in roughly the same beat a person would give — call it 800ms to 1.2s from the moment they stop talking. That total round-trip has to cover STT finalization, LLM inference, TTS first-byte, and network/SIP transport. STT doesn't get the whole budget. It gets maybe 300ms of it.

So the question isn't "which engine has lower word error rate." It's "which engine gives me usable transcripts fast enough that the other three stages still fit." A model that's 2% more accurate but adds 400ms of finalization latency is a worse voice-agent engine, full stop. Batch-accuracy benchmarks measure the wrong axis for this job.

Benchmark Setup: Telephony-Grade Audio, WER, and Why Clean-Audio Scores Lie

Most published WER numbers come from LibriSpeech or similar — read speech, 16kHz, studio-clean. Phone calls are the opposite: 8kHz narrowband (or G.711 μ-law), codec artifacts, background noise, cross-talk, accents, and callers who mumble their account number.

When you re-run the same models on telephony audio, the gap moves:

• Whisper large-v3 on clean 16kHz: ~5-8% WER. On 8kHz noisy telephony: often 12-18% WER, and it hallucinates fluent-but-wrong text on silence and noise — a known failure mode that's dangerous when the transcript feeds an LLM.
• Deepgram Nova-2/Nova-3, tuned for phone audio and streaming: 8-13% WER on the same telephony set, and it degrades more gracefully — it tends to drop words rather than invent them.

The lesson: benchmark on your own audio. Pull 200 real recorded calls, hand-label them, and score both engines on that. The ranking you get will not match any vendor blog, including this one. If you take one thing away, take that.

Latency & Endpointing: First-Token, Finalization, and Barge-In

Three latency numbers matter, and only one shows up in marketing.

First-token latency — how fast partial transcripts start streaming back. Deepgram streams interim results in ~100-200ms. Whisper is a batch model; the community streaming wrappers (whisper-streaming, WhisperLive) chunk audio and re-decode, which means partials arrive in 500ms-1s+ and jitter under load.

Finalization latency — how long after the caller stops speaking before you get a stable final transcript you can hand to the LLM. This is gated by endpointing (voice-activity detection deciding the turn is over). Deepgram ships tunable endpointing (endpointing=300). With Whisper you're bolting on your own VAD (Silero, WebRTC VAD) and tuning it yourself — more control, more rope.

Barge-in — when the caller interrupts the agent mid-sentence, you need to detect speech and kill TTS in ~100-200ms or the agent talks over them. This is a streaming-partials + VAD problem. Native-streaming engines make it easy; batch Whisper makes it a project.

Endpointing tuning is the single highest-leverage latency knob in the whole stack — set it too aggressive and you cut callers off mid-word; too loose and you add 500ms of dead air to every turn. See our low-latency voice engine teardown for where this sits in the pipeline.

Self-Hosted Whisper: GPU Cost, Batching, and the Ops Burden Nobody Prices In

"Whisper is free" is the most expensive sentence in voice AI. The model weights are free. Running them in real time at call-center concurrency is not.

Real numbers for a self-hosted faster-whisper (CTranslate2) large-v3 deployment:

• GPU: An A10G or L4 handles roughly 8-15 concurrent real-time streams at large-v3 before latency degrades. At 100 concurrent calls you need ~8-12 GPUs. On-demand cloud A10G runs ~$1.00-1.30/hr; reserved lands lower. Call it $7,000-10,000/month of GPU to hold 100 concurrent streams, before redundancy.
• Batching tradeoff: Batching lifts throughput per GPU but adds latency — the exact thing you're protecting. Real-time voice caps how much you can batch, so your GPU utilization runs low (often 30-50%), which is money burning.
• Ops burden: model loading, warm pools to avoid cold-start, autoscaling on bursty inbound traffic, VAD tuning, hallucination filtering, GPU driver/CUDA maintenance, and 24/7 on-call when a node wedges mid-call. That's 0.5-1 FTE of platform engineering that never shows up in a per-minute comparison.

Amortized, self-hosted Whisper at 100 concurrent calls frequently lands at $0.006-0.012/min all-in — competitive on paper, but only once you've paid the engineering to make it reliable. Below serious volume, it's almost never worth it. Our hybrid sub-120ms ASR pipeline covers the batching-vs-latency math in depth.

Deepgram / Managed APIs: Cost-Per-Minute at Scale and Regional Routing

Managed STT flips the tradeoff: higher marginal cost, near-zero ops.

• Deepgram Nova streaming lists around $0.0077/min (Nova pay-as-you-go, streaming), with volume/committed-use discounts pulling it lower at scale.
• No GPUs, no warm pools, no CUDA. You get autoscaling, endpointing, and streaming for free-of-effort.
• Regional routing matters more than the sticker price. If your media terminates in Mumbai and your STT endpoint is us-east, you've just added 200-300ms of transatlantic round-trip to every partial. Managed vendors with regional endpoints (or self-hosted enterprise deploys near your media plane) erase that. For dual-market US-India stacks this is decisive — see our cross-border latency architecture.

The honest read: for most teams under ~50 concurrent calls, managed wins on total cost once you price the engineer. Above that, the math starts favoring self-hosting — if you have the platform team.

How STT Choice Cascades Into Total Round-Trip Latency

Here's a representative turn budget for an inbound call, target < 1000ms perceived response:

Same LLM, same TTS. The STT choice alone swings the caller's experience across the "feels human" line. That 450ms is the difference between an agent people talk to and one they hang up on. STT latency isn't a line item — it's a multiplier on everything downstream. This is why we treat it as a budget, not a benchmark. More on the full pipeline in beyond the wrapper.

Decision Matrix: Pick by Call Volume, Language Mix, and Latency SLA

• < 50 concurrent, English-heavy, tight SLA → Managed (Deepgram or peer). You cannot beat the effort-adjusted cost, and streaming/endpointing is solved.
• High volume (100+ concurrent), stable traffic, platform team on staff → Self-hosted Whisper (faster-whisper) starts winning on marginal cost. Budget the FTE honestly.
• Heavy multilingual / code-switching → Test both on your languages. Whisper's multilingual breadth is strong but WER varies wildly by language; managed engines vary too. Don't assume — measure. See the hidden multilingual tax.
• Regulated / data-residency (healthcare, India DLT, EU) → Self-hosted or a regional managed deploy, chosen by where your media legally has to stay.
• Hallucination-sensitive (transcript feeds an LLM that acts on it) → Weight Whisper's silence-hallucination against Deepgram's graceful degradation, and add guardrails regardless. Our stopping voice AI hallucination playbook applies directly.

The Bottom Line

Whisper vs Deepgram is not an accuracy contest — it's a latency-budget and ops-burden decision. Managed wins on effort and time-to-ship; self-hosted Whisper wins on marginal cost only at real scale and only with a platform team behind it. Either way: benchmark on your own telephony audio, tune endpointing hard, and account for the full round-trip. The transcript is just the first 300ms of a 1-second promise to the caller.

FAQ

Is Whisper more accurate than Deepgram in 2025? On clean 16kHz audio, Whisper large-v3 is very strong. On 8kHz noisy telephony — the audio a voice agent actually hears — the gap narrows or reverses, and Whisper's silence-hallucination is a real risk. Benchmark on your own recorded calls before deciding.

How much does it really cost to self-host Whisper? Not free. At 100 concurrent real-time streams, expect 8-12 GPUs ($7-10k/month) plus 0.5-1 FTE of platform engineering. All-in it lands around $0.006-0.012/min — competitive with managed only at scale.

Can Whisper do real-time streaming? Not natively — it's a batch model. Community wrappers (WhisperLive, faster-whisper + VAD) add streaming, but first-token latency (500ms-1s+) and jitter are worse than native-streaming APIs. Barge-in and endpointing become your problem to build.

Which is better for a contact-center voice agent? For most teams under ~50 concurrent calls, a managed streaming API (Deepgram or peer) with tuned endpointing and a regional endpoint. Above that, with a platform team, self-hosted Whisper starts winning on cost. The SLA, not the WER, should drive the call.

Build a voice agent that hits the latency budget

Finn ships production contact-center voice agents with the STT, endpointing, and regional routing already tuned to the sub-1s round trip — so you skip the GPU math and the 3am pager. See how Finn keeps voice agents under the latency line →