Voice Agent Latency: Where the 2–3 Second Delay Actually Lives in the Pipeline and How to Reduce It

TL;DR: The 2–3 second delay in voice agents is not random — it’s five sequential stages stacking up: VAD, STT, LLM, TTS, and network. Moving to a streaming architecture and tuning each component can cut perceived latency to under 500ms. Always measure P95 and P99, not averages.

A 2–3 second silence after you stop speaking feels broken. For voice agents, that pause is a conversion killer. Understanding where the delay lives lets you fix the right thing instead of guessing.

Here’s what you need to know:

• VAD endpointing adds 300–800ms just waiting for silence to confirm you stopped speaking.
• STT processing contributes 150–500ms depending on model and streaming configuration.
• LLM time-to-first-token (TTFT) ranges from 200ms to over 2 seconds as context grows.
• TTS generation adds another 100–500ms before the first audio byte plays.
• Network transport adds 50–200ms, sometimes 700ms on PSTN telephony.

Where Each Stage Loses Time

Every voice interaction runs through five stages: VAD, STT, LLM, TTS, and network transport. In a sequential architecture, these stack. VAD at 800ms, STT at 500ms, LLM at 1,000ms, TTS at 500ms, and network at 200ms adds up to 3 full seconds before the user hears anything.

Streaming architectures overlap these stages. The LLM starts processing partial transcripts while the user is still speaking. TTS begins synthesizing as soon as the first sentence fragment arrives. This parallel approach cuts perceived latency to 300–500ms.

“Streaming architecture is not optional… the single biggest latency optimization in voice AI is not model selection, hardware, or network routing. It’s eliminating sequential waiting.” — Tianpan.co

Component-Level Fixes That Actually Move the Number

VAD: The silence threshold is the real culprit, not VAD processing itself. Lowering it to 200–400ms reduces endpointing delay. ML-based VAD tools like Silero or Deepgram Flux use pitch, energy, and lexical signals to cut endpointing to 150–200ms and reduce false interruptions by roughly 45%.

STT: Streaming STT sends partial transcripts in real time, letting the LLM start before the utterance ends. Disabling optional features like punctuation and filler word detection shaves additional time. Quantized models like Whisper Tiny INT8 reduce inference by 40–60% on constrained hardware.

LLM: TTFT is often the biggest single bottleneck. Groq-hosted Llama variants can reach 50–150ms TTFT. GPT-4o-mini reduced median TTFT from 2.8 seconds to 0.9 seconds in one production telephony deployment compared to GPT-4 Turbo. Summarizing conversation history instead of passing the full context prevents P95 TTFT from climbing above 2.2 seconds as calls grow longer.

TTS: Standard neural TTS averages 300ms before the first audio byte. Optimized systems like Cartesia Turbo bring that to 40ms. Streaming TTS starts playback on the first sentence fragment, overlapping generation with LLM token output. Barge-in handling must cancel queued audio within milliseconds when a user interrupts.

Network: WebRTC delivers round-trip times of 20–50ms. PSTN telephony adds 150–700ms from carrier routing. Cross-region calls stack 50–100ms per hop. Persistent WebSocket or gRPC connections with DNS caching eliminate repeated TCP handshake costs.

A Real-World Example

In March 2026, engineer Mohd Mursaleen optimized Claritel’s voice agent by targeting specific components. Switching to streaming TTS removed 1.1 seconds. Adopting GPT-4o-mini and trimming LLM context saved 2.2 seconds. Lowering the VAD silence threshold from 800ms to 400ms cut another 400ms. Call completion rates jumped from 61% to 89%.

How to Measure Before You Optimize

“If you can’t produce a per-turn latency log for every single turn, you’re tuning blind.” — Pradeep Jindal, ByondLabs

Log timestamps at every stage: VAD end-of-utterance, STT completion, LLM first token, TTS first byte. The formula for perceived latency is EOU delay plus LLM TTFT plus TTS TTFB. Use OpenTelemetry for nested spans across components.

Never rely on averages. A 300ms average can hide the fact that 10% of users wait 1,500ms. Track P95 and P99. Latency under 500ms feels natural. Above 1,200ms, users repeat themselves or hang up. Above 3 seconds, abandonment rates exceed 40%.

Key Points

• Move from sequential to streaming architecture first. It delivers the largest single reduction in perceived latency.
• Target VAD silence thresholds of 200–400ms and use ML-based endpointing where possible.
• Choose LLM models optimized for TTFT and summarize context aggressively as conversations grow.
• Use streaming TTS with barge-in support to overlap generation with LLM output.
• Measure P95 and P99 per turn. Averages hide the tail latency that drives users away.

Ready to go further?

• Curious about the full story? Read the original blog post on our website for additional insights.
• Want more insights? Browse our entire blog library to stay updated on industry trends and expert tips.
• Explore real-world success stories: Check out our case studies to see how we’ve helped businesses grow.
• Looking for a reliable tech partner? Learn more about us on our homepage and discover how we can support your vision.
• Eager to build and launch fast? Visit our Rapid MVP Launch page to kickstart your product journey today.