Agents, lesson 16
Voice agents
Agents on a live call, where every pause is heard.
On a call, the silence before an agent answers is part of the answer. This lesson budgets that silence, then covers turn-taking and the mistakes speech adds. You need the loop from "Observe, decide, act", and time to first token from "First token and every token after" in Inference and Serving.
By the end, you can:
- Add up a voice agent's time to first audio, stage by stage, and compare it with a target.
- Start speech on the reply's first sentence, and stop it cleanly when the caller cuts in.
- Choose how to detect the end of a turn, and say what each choice costs.
- Cover a slow tool with a status line, and read back what you heard before acting on it.
Part 1. Where the wait comes from
Add up a voice agent's time to first audio, stage by stage, and compare it with a target.
Ring a support line run by an agent and ask "Where's my order?" Three stages sit behind the voice: speech to text writes down your words, the model replies in text, and text to speech reads it aloud. Each waits for the last, and you hear only silence.
Example 1.1
One turn's budget
Example numbers, all assumptions: a 500 ms silence wait before the agent decides you've finished, the final transcript 100 ms later, the model's first token in 350 ms, the speech engine's first audio in 150 ms, and the network 50 ms each way. When do you hear the agent?
- Wait out the silence500 ms
Until then it can't tell a finished question from a pause.
- Finish the transcript500 + 100 = 600 ms
Streaming speech to text wrote words as you spoke; this is its final version.
- Wait for the first token600 + 350 = 950 ms
Network, queue and prefill, from "First token and every token after".
- Make the first audio950 + 150 = 1,100 ms
As if the speech engine starts on the first token. Part 2 adds the real wait.
- Cross the network1,100 + 100 = 1,200 ms
Your audio up to the server, and the reply's audio back.
Answer1,200 ms, and 500 of it, 42%, is the silence wait.
Time to first audio
The time from the caller's last word to the first sound of the reply. In a pipeline it's the sum of the stages, so log each one, and watch the median and the slowest turns.
silence wait + transcript + first token + first audio + network
What should it be? People reply after about 200 ms on average, and gaps of 700 ms or more tend to come before an unwelcome answer, such as a no (Levinson and Torreira, 2015). No pipeline gets near 200 ms, so this lesson takes 1 second as an example target, not a standard.
Some providers offer one model that hears and answers in audio over a live connection, such as OpenAI's Realtime API and Google's Gemini Live API. Nothing waits on a separate transcript or speech engine, but there's no text in the middle to read, check and log.
Your turn
The same agent on a phone line, in example numbers: a 500 ms silence wait, transcript 150 ms, first token 400 ms, first audio 200 ms, and the phone network 150 ms each way. What's its time to first audio?
Part 2. Speak while the model writes
Start speech on the reply's first sentence, and stop it cleanly when the caller cuts in.
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Speak while the model writes
- Knowing the caller has finished
- Tools and mistakes on a live call
Agents overviewSpotted a mistake? Tell us.