When you talk to another person on the phone, responses come within a few hundred milliseconds. That cadence is deeply ingrained — humans are remarkably sensitive to conversational timing. Add even half a second of unexpected delay and the conversation starts to feel off. Add a full second and people start talking over each other, asking "are you there?", or assuming the line dropped.

This is the latency challenge for AI voice agents. The AI must hear what you said (STT), figure out what to say (LLM), and say it aloud (TTS) — all within a window that feels natural. Is that possible in 2026? Yes. Is it easy? No. This article explains how latency engineering works for voice agents, what "fast enough" actually means, and how modern systems achieve sub-second response times.

What latency feels natural

Conversational timing research gives us clear benchmarks:

0–300ms: Feels instantaneous. This is the range for simple acknowledgments ("uh-huh", "right") that signal active listening. Humans produce these reflexively.
300–800ms: Feels natural. This is the normal range for a considered response in conversation. The listener perceives the speaker as "thinking" — which is appropriate.
800ms–1.5s: Feels slow but tolerable. The listener notices the delay but does not assume a problem. Equivalent to a thoughtful pause or someone checking their notes.
1.5s–3s: Feels awkward. The listener starts to wonder if something is wrong. They may repeat their question or say "hello?"
3s+: Feels broken. The listener assumes a technical problem and may hang up.

For AI voice agents, the target is 300–800ms from the end of the caller's utterance to the beginning of the agent's response. This is called "time to first byte" (TTFB) or "response latency."

Achieving this consistently — not just on average, but at the 95th and 99th percentile — is the core challenge of voice AI engineering.

The latency budget

Response latency is the sum of four sequential steps. Understanding each step's contribution is essential for optimization.

Step 1: End-of-turn detection (50–300ms)

Before the AI can start processing, it needs to determine that the caller has finished speaking. This is harder than it sounds. A pause might mean the caller is done, or it might mean they are thinking of the next word.

Voice Activity Detection (VAD) algorithms listen for silence duration, falling intonation, and completed sentence patterns. The tradeoff is direct:

Aggressive VAD (short silence threshold): Faster response, but risks cutting off the caller mid-sentence.
Conservative VAD (long silence threshold): Fewer interruptions, but adds latency to every turn.

Most production systems use a silence threshold of 300–500ms, with additional heuristics for sentence completion. The best systems also use semantic end-of-turn detection — the LLM itself predicts whether the transcript so far constitutes a complete turn.

Step 2: Speech-to-text (100–500ms)

The caller's audio must be transcribed to text. Modern streaming STT services process audio in real time, producing partial transcripts as the caller speaks and a final transcript within 100–300ms of the end of speech.

Key factors affecting STT latency:

Streaming vs. batch. Streaming STT begins processing as audio arrives, producing results incrementally. Batch STT waits for the complete utterance. Streaming is essential for low latency.
Provider choice. Deepgram, Google Cloud STT, and AssemblyAI offer streaming with final transcript latency of 100–300ms. Some providers are faster for specific languages or accents.
Network latency. The round trip between the call server and the STT service. Co-locating these services in the same region saves 20–50ms.

Step 3: LLM inference (200ms–2s+)

The LLM reads the conversation history and the latest transcript, then generates a response. This is typically the largest contributor to total latency.

LLM inference time depends on:

Model size. Smaller, faster models (GPT-4o-mini, Claude Haiku, Gemini Flash) respond in 200–500ms for short outputs. Larger models (GPT-4o, Claude Opus) take 500ms–2s.
Prompt length. Longer conversation histories and larger system prompts increase processing time. A 2,000-token prompt processes faster than a 10,000-token prompt.
Output length. The LLM generates tokens sequentially. A 50-token response is produced 5x faster than a 250-token response. Voice agent prompts should instruct the model to be concise.
Streaming. Streaming LLM output sends tokens as they are generated rather than waiting for the complete response. The first token arrives in 200–400ms; the full response may take 1–2s, but TTS can begin on the first sentence immediately.
Provider load. LLM API latency varies significantly by time of day and current demand. Peak hours can add 200–500ms.

Step 4: Text-to-speech (100–500ms)

The LLM's text response is converted to audio. Like STT, streaming is critical.

Streaming TTS begins generating audio as soon as the first sentence of LLM output is available. First audio byte arrives within 100–200ms of receiving text.
Neural TTS (ElevenLabs, Cartesia, Play.ht) produces higher-quality speech but is slightly slower than older parametric TTS systems.
Sentence-level chunking. The system feeds the TTS one sentence at a time from the streaming LLM output, allowing audio playback to begin while the LLM is still generating subsequent sentences.

Total budget

Adding up the best-case and worst-case for each step:

Step	Best case	Typical	Worst case
End-of-turn detection	50ms	200ms	500ms
STT (streaming)	100ms	200ms	500ms
LLM (first token)	150ms	400ms	1500ms
TTS (first audio byte)	50ms	150ms	400ms
Total	350ms	950ms	2900ms

The typical case (950ms) falls within the acceptable range. The best case (350ms) feels instantaneous. The worst case (2900ms) feels broken. The engineering challenge is keeping the system in the typical-to-best range consistently.

How streaming makes it work

The key insight in voice AI latency engineering is that nothing needs to wait for anything else to fully complete. Every step can stream into the next:

STT streams partial transcripts while the caller is still speaking. The LLM can begin processing before the final transcript arrives.
The LLM streams tokens as they are generated. TTS can begin synthesizing the first sentence while the LLM is still producing the second.
TTS streams audio chunks to the caller as they are generated. The caller starts hearing the response while TTS is still synthesizing later parts.

This pipelined architecture means the caller hears the first word of the agent's response hundreds of milliseconds before the full response has been generated. The experience feels like a natural conversational pause followed by a fluent response — even though the underlying system is still computing the rest of the answer.

Without streaming, the system would need to complete each step sequentially, resulting in 2–5 second response times. With streaming, effective latency drops to 500–800ms for most interactions.

Edge deployment and geographic optimization

Network latency is the silent tax on every voice AI interaction. Each network hop adds 5–50ms of round-trip time.

The problem: If your caller is in Dallas, your call server is in Virginia, your STT is in Oregon, your LLM is in San Francisco, and your TTS is in London, every step involves a cross-country or cross-continent round trip. These add up to 100–300ms of pure network overhead.

The solution: Edge deployment places voice AI infrastructure close to callers:

Call servers in multiple geographic regions, routing callers to the nearest one.
STT and TTS co-location with the call server or in the same region.
LLM inference at the edge where possible (smaller models), or via dedicated capacity in the nearest cloud region for larger models.
Connection pooling and keep-alive to eliminate the overhead of establishing new connections for each request.

Geographic optimization alone can save 50–150ms of response time — the difference between "feels natural" and "feels slow."

Turn-taking design

Latency engineering is not purely about speed. How the agent manages conversational turns has an equally large impact on perceived responsiveness.

Filler phrases and acknowledgments

Human speakers use filler phrases ("Let me check...", "Sure...", "One moment...") to signal that they are processing a request. AI agents can do the same:

When the request requires a function call (database lookup, API call), the agent immediately says a filler phrase while the operation executes in the background.
When the LLM is taking longer than expected, a brief acknowledgment ("Got it...") buys time without creating awkward silence.

These fillers do not actually reduce processing time, but they reduce perceived latency dramatically. A 1.5-second wait feels acceptable after "Let me look that up for you" but uncomfortable in silence.

Predictive response initiation

Some systems begin generating a response before the caller has fully finished speaking, based on the partial transcript. If the system is 90% confident it knows what the caller is asking after their first sentence, it can begin LLM inference immediately rather than waiting for the caller to finish.

This is aggressive and risks generating the wrong response if the caller changes direction. But for common, predictable requests ("What are your hours?" "I need to check my order status"), it can shave 200–400ms off response time.

Barge-in handling

When a caller interrupts the agent mid-sentence, the system must:

Immediately stop TTS playback.
Begin STT processing of the interruption.
Cancel any in-progress LLM generation.
Process the interruption as a new turn.

Fast barge-in handling (under 200ms to stop and switch) is essential for natural conversation. Slow barge-in creates the maddening experience of talking over an AI that will not stop speaking.

Comparing AI latency to human latency

A useful frame of reference: how fast are human agents?

Hold time before answering: 30 seconds to 15+ minutes. AI agents answer instantly.
Response latency during conversation: 500ms–2s. Comparable to AI agents.
System lookup time: 5–30 seconds (navigating CRM, searching knowledge base). AI agents perform lookups in 200–500ms.
After-call processing: 30s–5min. AI agents process in real time.

In total interaction time, AI agents are typically 30–60% faster than human agents. The latency comparison is often framed as "AI is slower than humans" because people compare the AI's 800ms response time to a human's 500ms response time. But they forget the 8-minute hold time before the human answered.

For the customer, the experience of calling, getting an immediate answer, and resolving their issue in 90 seconds feels dramatically faster than calling, waiting on hold for 8 minutes, and resolving the same issue in 3 minutes.

What "too slow" actually looks like

When does latency become a real problem in practice?

Consistent 2s+ response times. If every exchange takes two or more seconds, the conversation feels like talking to someone on a satellite phone. Callers tolerate one slow response (they assume you are checking something) but not an entire conversation of them.
Latency variance. Inconsistent timing is worse than consistently slow. If most responses come in 500ms but every fourth response takes 3 seconds, the unpredictability is jarring.
Latency on simple acknowledgments. "Yes," "I see," and "Got it" should come in under 500ms. When confirmations are slow, the caller feels unheard.
Compounding latency. Long conversations accumulate context, increasing LLM processing time. A system that responds in 600ms at the start of a call but 2 seconds by minute five has a context management problem.

The state of the art in 2026

The best voice AI platforms in 2026 achieve:

P50 response latency: 400–600ms (median call).
P95 response latency: 800–1200ms (95th percentile).
P99 response latency: 1500–2500ms (99th percentile).

These numbers are achievable with streaming STT, streaming LLM output, streaming TTS, edge deployment, and proper model selection. They represent a 3–5x improvement over 2024-era systems.

The remaining frontier is not raw speed but consistency — closing the gap between P50 and P99 so that no caller experiences a noticeable delay. This requires better capacity planning, smarter model routing (using faster models for simple responses, larger models only when needed), and continued infrastructure optimization.

AI is not too slow for real phone calls. It was too slow in 2023. In 2026, the latency challenge is an engineering problem with known solutions — not a fundamental limitation of the technology.

Frequently Asked Questions

What is the minimum acceptable response latency for a voice agent?

The target is 300–800ms time to first audio byte. Responses under 300ms feel instantaneous. Responses between 800ms and 1.5s are tolerable but noticeable. Consistent response times above 1.5s create a poor caller experience.

Does using a more powerful LLM always mean higher latency?

Generally yes — larger models produce higher-quality responses but take longer to generate them. However, streaming output mitigates this because the caller hears the first word before the full response is generated. The practical tradeoff is between response quality and time-to-first-token, not response quality and total generation time.

Can I reduce latency by using a smaller, fine-tuned model instead of a frontier model?

Yes. A small model fine-tuned on your specific domain and conversation patterns can match or exceed a frontier model's quality for your use case while responding 2–5x faster. This is one of the most effective latency optimizations for high-volume deployments with well-defined conversation patterns.

How does latency change as the conversation gets longer?

LLM processing time increases roughly linearly with context length. A 20-turn conversation has approximately 2x the LLM latency of a 5-turn conversation due to the longer prompt. Production systems manage this through context summarization (condensing earlier turns into a summary), sliding windows (dropping the oldest turns), and model routing (switching to a faster model for longer contexts).

Is AI Too Slow for Real Phone Calls? Latency Engineering for Voice Agents