A voice agent is not a chat agent with a microphone bolted on. It is a real-time system whose success is measured in milliseconds and interruptions — whether the reply starts before the silence gets awkward, whether the agent stops talking the instant you cut in, whether it hears your accent over the café noise. ADK gives you the runtime for exactly this: run_live and the LiveRequestQueue turn the framework from request/response into a continuous audio conversation, and the Gemini Live API supplies native speech-in, speech-out with the paralinguistic signal a transcribe-then-prompt pipeline throws away. This piece is the builder’s view of that stack: the audio formats you actually feed it, how voice activity detection decides when you have finished a sentence, how barge-in really works once you own the client buffers, the latency budget that separates ‘conversation’ from ‘IVR menu’, and how the same session maps onto a browser tab or a phone call. The companion pieces on the streaming architecture and on multimodal content cover the systems and the media typing; here we stay on the wire where the audio lives.
Native audio vs the cascaded ASR→LLM→TTS pipeline
The first decision defines everything downstream: does the model hear and speak, or does it read and get read to? The classic voice stack is a cascade — speech recognition (ASR) turns audio into text, an LLM prompts on that text, and text-to-speech (TTS) turns the answer back into audio. It is modular and every stage is swappable, but it stacks three latencies serially and, worse, it discards paralinguistics at the very first hop: tone, hesitation, sarcasm, urgency, and emotion are gone the moment audio becomes a bare string.
A native audio model — the mode the Gemini Live API exposes and ADK drives through run_live — collapses the stack: speech goes in, speech comes out, one model, one session. It can hear that you sound frustrated and soften its reply; it can begin speaking a beat after you stop because there is no TTS round trip to wait on. The trade is control: you no longer swap in a boutique ASR tuned to medical vocabulary, and voice selection is constrained to what the model offers. A pragmatic middle path also exists — native audio input for understanding, but your own TTS on text output when you need a specific branded voice or on-prem synthesis. Pick deliberately, because it dictates your latency floor and your quality ceiling.
The audio pipeline: sample rates, PCM, and framing
Voice bugs are usually audio-format bugs, so get the numbers right first. The Gemini Live API expects 16-bit signed PCM, little-endian, 16 kHz, mono on the way in, and emits audio at 24 kHz on the way out. Those two rates being different is the single most common source of chipmunk-voice and slow-motion-voice bugs: you must resample the microphone to 16k before sending and play the response back at 24k, not at whatever your capture device defaults to.
Audio is streamed in frames, typically 20–100 ms of samples per chunk. Smaller frames mean lower latency but more overhead; larger frames the reverse. In the browser you capture with the Web Audio API (AudioWorklet for low-latency PCM access), downsample, and ship raw bytes; on a phone line you are handed 8 kHz µ-law and must transcode. ADK does not hide this from you — it forwards whatever bytes you push into the queue — so the resampling, the mono downmix, and the little-endian byte order are your responsibility. A quick sanity check before debugging anything ‘the model can’t hear me’: capture a few seconds, write it to a .wav with the exact header you are sending, and listen. Nine times out of ten the format, not the model, is wrong.
run_live and the LiveRequestQueue: the voice loop
Turn-based ADK calls Runner.run_async and gets a stream of events for one turn. A voice agent instead calls run_live, which never ‘returns’ in the turn sense — it opens a live model session and stays open. Its upstream half is the LiveRequestQueue: the pipe you push into. You send audio bytes with the realtime path (send_realtime) and text or control signals with send_content; the model consumes them continuously rather than waiting for a completed turn.
The shape that falls out is two concurrent pumps. One asynchronous task drains your client transport (the WebSocket) and pushes each audio frame into the LiveRequestQueue. A second task iterates the async event stream run_live returns and forwards each event — audio chunks, partial text, transcriptions, interruption signals — back down to the client. Neither blocks the other, which is the whole point: the user can speak while the agent is still talking, and both directions move at once. This bidirectional, always-open loop is the architectural heart of ADK streaming; the companion streaming article dissects the event pumps, session resumption, and mid-stream tool calls in depth. Here it is enough to know that the queue is your microphone-to-model conduit and the event stream is your model-to-speaker conduit.
Integrating the Gemini Live API
Under run_live, ADK maintains a persistent connection to the Gemini Live API — a stateful, low-latency socket built for bidirectional audio, distinct from the standard request/response generateContent endpoint. You configure it through the agent’s run configuration rather than by touching the socket directly. The knobs that matter for voice are the response modality (audio versus text — you generally pick one per session), the voice (the Live API ships a set of prebuilt voices you select by name), and whether input and output transcription are enabled.
Two practical notes. First, the live session runs its own voice activity detection on the audio you stream — you are not expected to do endpointing before sending; you send continuously and the model decides when the user has yielded the floor. Second, live connections have hard duration limits and drop on network changes, so the API issues session-resumption handles you reconnect with to restore context. Model choice matters here too: Google offers native-audio dialog models tuned for expressive, natural speech and separate half-cascade configurations that pair native audio input with TTS output for steadier tool-use behavior. Choose based on whether expressiveness or tool-calling reliability is your priority — you cannot always maximize both in the same model.
Voice activity detection and endpointing
Voice activity detection (VAD) answers the question every voice system must answer many times a second: is the user speaking right now, and have they finished? Get it wrong toward ‘too eager’ and the agent interrupts you mid-thought after a breath; wrong toward ‘too patient’ and there is a dead pause after every sentence that makes the agent feel slow and dim. This trade — snappy versus not cutting people off — is the central tuning problem of conversational voice.
With the Gemini Live API the VAD runs server-side on the audio you stream: it detects speech start, speech end, and produces the endpointing decision that a user turn is complete, which is what triggers the model to respond. It is generally automatic, but exposes sensitivity and silence-duration controls so you can bias it for your acoustic environment — a noisy call center wants different thresholds than a quiet desktop. Some deployments still run a cheap client-side VAD in addition, purely as an optimization: gating the microphone so you do not pay to stream pure silence or background chatter upstream, and to drive a ‘listening’ UI indicator. The authoritative turn decision, though, belongs to the model that can weigh linguistic context, not just energy levels — a rising intonation (‘so I was thinking…’) is not a finished turn even after a long pause.
Barge-in and interruption handling
Barge-in — the user talking over the agent — is the single feature that most separates a natural voice agent from an answering machine. Humans overlap constantly; an agent that ploughs through its whole sentence after you have clearly cut in reads as a robot instantly. When the model’s VAD detects user speech during playback, ADK surfaces an interruption event: the model halts generation, and the event tells you where its utterance was cut so the session records what the user actually heard, not what was generated.
The gotcha that bites every team once: flushing the client buffer is your job, not the model’s. By the time the interruption event arrives, your client may already hold one to three seconds of the agent’s audio queued for playback. If you do not immediately discard that buffer, the agent keeps talking for seconds after it should have stopped — the worst possible feel. So the interruption handler must stop the audio source and clear the queue at once. The second, subtler requirement is acoustic echo cancellation (AEC): without it, the microphone picks up the agent’s own voice from the speakers and the VAD triggers on the agent interrupting itself. In the browser, getUserMedia’s echoCancellation constraint handles the common case; on speakerphone and telephony you lean on hardware or network AEC.
The latency budget for natural conversation
Conversation has a felt deadline. Human turn-taking gaps cluster around 200–300 ms; a voice agent that starts replying within roughly 500–800 ms of you finishing feels conversational, and past about 1.5 s it feels like an IVR system. That budget is not the model’s inference time alone — it is the whole loop, and every hop spends part of it.
| Hop | Rough budget | Where it goes |
|---|---|---|
| Endpointing delay | ~100–300 ms | VAD waiting to be sure you stopped |
| Network up | ~20–100 ms | Client to server to Live API |
| Model first token | ~200–500 ms | Time-to-first-audio-chunk |
| Network down + buffer | ~20–100 ms | First chunk to speaker |
The design consequences are concrete. Measure time-to-first-audio, not total response time — the user only cares when speech starts. Start playback on the first chunk; never wait for the full utterance. Keep endpointing as tight as accuracy allows, because it is dead air the user feels directly. And put the agent server close to both the user and the Live API region — a trans-oceanic hop can eat your entire budget before the model has done any work. Slow tools are the classic budget-killer: cover them with a spoken acknowledgement (‘let me check that’) so silence never stretches past a second.
Transcription: captions and the textual record
Even when the agent thinks in audio, you almost always want text on the side. The Live API can produce input transcription (what the user said) and output transcription (what the agent said), streamed as events alongside the audio. Enable both and three things become possible: live captions in the UI for accessibility and noisy environments, a searchable textual log of the conversation, and a session record that stays meaningful even though the payload was speech.
That last point is architecturally important. A voice session logged only as audio blobs is nearly useless for debugging, analytics, evaluation, or compliance. The transcription events let ADK write a textual event history — the same event-sourced session record the rest of the framework uses — so a conversation that happened in voice can later be resumed, audited, or even continued in text mode without losing the thread. There is a subtlety worth flagging: because of barge-in, the output transcript should reflect what the user heard, truncated at the interruption point, not the full sentence the model started generating. If you log the generated text rather than the interruption-adjusted text, every downstream turn and every analytics query inherits a transcript that lies about what was actually said.
Transport: WebSocket, WebRTC, and SSE
Between the client and your ADK server you need a bidirectional, low-latency channel, which immediately rules some options in and out. Server-Sent Events (SSE) is one-way (server to client) and fine for streaming a text agent’s tokens down, but it cannot carry the user’s microphone up, so it is not a voice transport on its own. WebSocket is the common choice: full-duplex, widely supported, and simple to pair with an ASGI server like FastAPI — you accept the socket, spin up the two pumps, and relay binary audio frames both ways.
WebRTC is the heavier but higher-quality option. It is built for real-time media: it brings jitter buffers, packet-loss concealment, adaptive bitrate, and crucially built-in echo cancellation and noise suppression — exactly the audio problems you would otherwise solve by hand. The cost is operational complexity (ICE, STUN/TURN, signaling). A useful rule of thumb: a controlled desktop or in-app experience on good networks does well on WebSocket; a consumer mobile product on flaky cellular, or anything where audio quality under packet loss is a product requirement, justifies WebRTC. Either way the client’s job is the same — capture and resample the mic, stream frames up, and play chunks down the instant they arrive with a short jitter buffer to smooth network unevenness.
Beyond audio: video and screen sharing
The Live API is not audio-only; the same session can accept a video stream — camera frames or a shared screen — alongside the microphone, which turns a voice assistant into something closer to a sighted collaborator. You push image frames into the LiveRequestQueue on the realtime path at a modest frame rate (you do not need thirty frames a second; one every second or two is often plenty for grounding a conversation), and the model reasons over audio and vision jointly.
The use cases are compelling: ‘what am I looking at?’ pointed at an appliance, a cooking assistant watching your pan, a support agent watching the user’s screen and talking them through a fix in real time. The engineering caution is cost and bandwidth — video frames are far heavier than audio in both tokens and network, so you sample sparingly and drop frames under pressure rather than backing up the audio path, which must stay real-time. How media is typed as content parts, how large media lives in artifacts, and the token economics of images are the subject of the companion multimodal article; the point here is simply that voice and vision share one live session, and the audio loop takes priority when you have to shed load.
A live streaming session: code sketch
The following sketch shows the two-pump structure end to end — a FastAPI WebSocket endpoint that bridges a browser to an ADK live session. Names follow current ADK streaming conventions; treat it as the shape, and check exact signatures against your installed version.
from fastapi import FastAPI, WebSocket
from google.adk.runners import Runner
from google.adk.agents import LiveRequestQueue
from google.adk.agents.run_config import RunConfig
from google.genai import types
app = FastAPI()
runner = Runner(agent=voice_agent, app_name="assistant",
session_service=session_service)
@app.websocket("/ws/{user_id}")
async def live_endpoint(ws: WebSocket, user_id: str):
await ws.accept()
session = await session_service.create_session(
app_name="assistant", user_id=user_id)
queue = LiveRequestQueue()
run_config = RunConfig(
response_modalities=["AUDIO"], # speak back
# enable input + output transcription for captions/logging
)
# downstream pump: model events -> client
async def to_client():
async for event in runner.run_live(
session=session, live_request_queue=queue,
run_config=run_config):
if event.interrupted: # barge-in!
await ws.send_json({"type": "flush"})
for part in (event.content.parts if event.content else []):
if part.inline_data: # 24kHz PCM out
await ws.send_bytes(part.inline_data.data)
elif part.text: # transcript delta
await ws.send_json({"type": "text", "t": part.text})
# upstream pump: client mic frames -> queue
async def from_client():
async for frame in ws.iter_bytes(): # 16kHz PCM in
queue.send_realtime(types.Blob(
mime_type="audio/pcm;rate=16000", data=frame))
import asyncio
await asyncio.gather(to_client(), from_client())
The two coroutines are the whole idea: from_client never blocks to_client, so audio flows both ways at once, and the event.interrupted branch is where barge-in becomes a flush instruction the browser obeys by dumping its playback buffer.
Use case: voice assistants
The archetypal voice agent is the hands-free assistant — in a car, on a smart speaker, in a wearable, or as an accessibility layer over an app. What these share is that voice is the primary interface, not a convenience, so the bar for naturalness is high: fast turn-taking, clean barge-in, and graceful handling of the messy realities of speech — half-sentences, ‘um’, corrections mid-utterance (‘set a timer for ten — no, fifteen minutes’).
ADK earns its place here because a real assistant is not just a talking LLM — it does things. The same tool-calling, session, state, and callback machinery that powers a text agent applies unchanged inside run_live: the assistant calls a set_timer tool, reads and writes session state to remember context across turns, and gates a sensitive action (‘unlock the door’) behind a before_tool callback — exactly as it would in text. The voice layer is a transport over that agent, not a different agent. This is the payoff of building on ADK rather than wiring the Live API by hand: your business logic, tools, and guardrails are transport-agnostic, so the same agent can serve a chat box and a microphone with the policy code written once.
Use case: call centers and telephony
The highest-value voice agents often live on a phone line — automating call-center front lines, appointment booking, order status, tier-one support. Telephony bends several of the earlier assumptions. Phone audio is 8 kHz µ-law (G.711), narrowband and noisier than a browser mic, so you transcode to the 16 kHz PCM the Live API wants and accept that recognition on a bad mobile connection is genuinely harder. The transport is usually a telephony provider (Twilio, a SIP trunk, or a CPaaS) that streams call audio to your server over its own media socket, which you bridge into the LiveRequestQueue just as you would a browser WebSocket.
Two call-center realities deserve first-class design. First, barge-in is non-negotiable — callers expect to interrupt a menu or a bot the instant they know what they want, and echo cancellation over a speakerphone is harder, so budget for it. Second, escalation to a human must be clean: when the agent hits its limit or the caller asks for a person, it hands off with the transcript and collected context intact — which is exactly why the input/output transcription and the durable session record matter here more than anywhere. Add DTMF (keypad) handling for the ‘press 1’ fallbacks and strict latency discipline, because dead air on a phone call feels even longer than it does on a screen.
Production concerns: cost, connections, and testing
Voice changes the economics and the failure modes, so a few operational habits separate a demo from a deployment. Billing is per-minute, not per-token — an open audio session bills continuously whether or not anyone is speaking, so idle detection and session timeouts are real cost levers, and knowing when to close the live socket and drop back to cheaper turn-based text is an architectural decision, not an afterthought.
Connection lifecycle is the other constant: live sessions have hard duration caps and die on network changes, so you build reconnection around the session-resumption handle from day one — the goal is that a caller walking from wifi to cellular hears a half-second gap, not amnesia. Instrument per-hop latency (endpointing, network, first audio chunk) so you can see which hop blew the budget, and track cost per minute alongside it. Finally, test with real audio: a voice agent that passes on clean studio recordings can fall apart on accented, fast, overlapping, or noisy speech, and on the specific acoustics of a car or a call center. Build evaluation sets from real recordings — interruptions, cross-talk, background noise included — because those are the conditions that break voice agents, and they are invisible to a text test suite.
run_live plus the LiveRequestQueue turn the framework into a continuous, bidirectional conversation, and the Gemini Live API supplies native speech-in and speech-out that a cascaded ASR→LLM→TTS pipeline cannot match on latency or nuance. Build it as two never-blocking pumps — mic frames up into the queue, model audio and transcription events down to the client — and sweat the audio details the model cannot fix for you: resample to the 16 kHz-in / 24 kHz-out PCM it expects, cancel echo, and above all flush the client buffer the instant a barge-in event fires, because that is what makes interruption feel human. Hold the whole loop inside a 500–800 ms time-to-first-audio budget, log the interruption-adjusted transcript as your durable record, and pick WebSocket or WebRTC to match your network. Because the voice layer is just a transport over an ordinary ADK agent, your tools, state, and guardrails are written once and serve a browser tab, a smart speaker, or a phone call alike — from voice assistants to call-center automation.