Buyers don't say "your latency is too high." They say "it feels robotic" or "it's a little awkward" — and then they don't sign. Turn-taking latency is the most common reason a technically capable voice agent feels wrong, and it's invisible on a feature sheet. Here's where the milliseconds actually go and the concrete moves that get you reliably under 800ms.
What latency actually is here
The number that matters is response latency — the gap between the caller finishing their sentence and the agent starting to speak. Humans expect this to be roughly 500ms in natural conversation. Past about 1 second it reads as "slow"; past 1.5s it reads as "broken" or "is it still there?" The other killer is interruption handling (barge-in) — what happens when the caller talks over the agent.
Where the milliseconds go
The pipeline is a chain, and latency is cumulative:
- Endpointing / VAD — detecting the caller actually stopped talking (not just paused). Aggressive endpointing feels snappy but cuts people off; conservative feels laggy. This is the biggest tunable lever.
- STT (speech-to-text) — transcribing what they said. Streaming STT that emits partials is far faster than waiting for a final transcript.
- LLM inference — generating the response. Token generation time + time-to-first-token. Model choice and prompt length both matter here.
- TTS (text-to-speech) — synthesizing the reply. Streaming TTS that starts speaking the first chunk while generating the rest is essential.
- Network / telephony — the transport round-trips, especially on cellular.
Every stage that waits for the previous one to fully finish instead of streaming adds dead air.
The moves that actually cut it
Tune endpointing first. This single setting is where most "laggy" agents live. Shorten the silence threshold for fast talkers, but add smart punctuation/semantic endpointing so it doesn't interrupt someone mid-thought. On Vapi/Bland/Retell this is exposed — spend real time here, it's the highest-ROI knob.
Stream everything. Streaming STT (partials), streaming LLM output, streaming TTS. The agent should begin speaking the start of its answer before it has finished generating the end. If any stage is non-streaming, you've capped your floor.
Shorten the prompt and the response. A bloated system prompt slows time-to-first-token. Long agent responses give the caller more chance to interrupt and feel talked-at. Tight prompts, concise replies — faster and more natural.
Pick the model for speed where it's safe. A slightly smaller/faster model for routine turns, escalating to a stronger one only for complex intents, keeps the common case snappy. Don't pay heavyweight inference time on "what are your hours."
Use filler/acknowledgment tokens. A brief "sure—" or "let me check that" while the real response generates masks latency the way a human's "uh, yeah" does. Used sparingly, it makes 900ms feel like 400ms.
Nail barge-in. The agent must stop talking the instant the caller speaks and actually listen — not finish its sentence. An agent that plows through interruptions feels deaf, and deaf feels robotic.
How to test it like a buyer will
- Call from a cell phone, not your laptop softphone. Cellular adds real latency and compression your office WiFi hides.
- Interrupt it three times. Mid-sentence. Does it stop and listen, or talk over you?
- Ask a fast follow-up right after it answers. Is the turn snappy or is there a beat of dead air?
- Time the gaps with a stopwatch on a recording if you have to. Under ~800ms typical, with clean barge-in, is the bar.
The bottom line
You can have the smartest agent in the niche and lose every deal to a competitor whose agent is dumber but feels alive. Conversational feel is mostly latency and interruption handling, and both are tunable. Fix the milliseconds before you add a single feature — it's the cheapest, highest-impact work you'll do on any deployment.