Latency

What does latency mean?

In general, latency means a delay: the time between a cause and its effect. In networking it is how long data takes to travel; in gaming it is lag. In AI, Anthropic's documentation defines it as the time it takes for the model to process a prompt and generate an output. "No latency" in marketing usually means latency low enough that people do not notice it, not zero.

How latency is measured

The analogy: ordering at a restaurant. Time to first token is how long until the first plate arrives; total latency is how long until the whole meal is on the table.

The precise version uses a few measurements:

• Time to first token (TTFT): the time from sending the prompt until the model produces the first token of its reply. It matters most when the answer is streamed to the user.
• Total response time: the time until the full answer is complete, which grows with the number of output tokens.
• Baseline latency: a general measure of a model's speed, separate from the size of the input and output.

What affects latency

Anthropic lists model size, hardware, network conditions, and the complexity of the prompt and response as factors. In practice, the main levers are choosing a faster model for speed-critical tasks, keeping prompts and outputs as short as the job allows, and streaming responses so people see the answer begin right away. Reasoning models that think before answering trade extra latency for accuracy.

Why latency matters for voice agents

In a text chat, a two-second pause is fine. On a phone call, it feels like the line went dead. A voice agent runs a loop on every turn: speech-to-text transcribes the caller, an LLM decides what to say and may call a tool, and text-to-speech speaks the reply. Each step adds delay, so voice systems lean hard on fast models, streaming, and short replies to keep the conversation natural.

A real example of latency

A caller asks a plumbing company's phone agent, "Can someone come out tomorrow?" If the agent waits for the full reply to be written before speaking, the caller hears silence. If the reply is streamed, the voice can start with "Let me check tomorrow's schedule" while the calendar lookup runs. Same model, much lower perceived latency.

Related terms

• Streaming: sending the reply as it is generated, the biggest lever on perceived latency.
• Inference: the model run whose duration latency measures.
• Token: more output tokens means more time to finish.
• Reasoning model: more accurate on hard problems, usually slower.

Common misconceptions about latency

• Latency is just internet speed. The network is one part; model size, prompt length, and output length often matter more.
• The fastest model is always the right one. Speed gains are not worth it if accuracy drops below what the job needs; Anthropic advises getting a prompt working well first, then optimizing latency.
• Streaming makes the model faster. It does not shorten total generation time; it shows the answer sooner, which feels faster.

Latency in practice

Latency is where phone agents succeed or fail. See how the whole call loop works in our guide to what an AI voice agent is, and read the streaming entry for the technique that does the most to hide delay.