Why architecture matters here
Voice agents live or die by a number: response latency around 500–800ms feels like conversation; beyond 1.5s feels like an IVR menu. That budget must cover audio capture, network, model inference, and playback start — which is why the architecture matters more than the model. A pipeline that transcribes fully, then prompts an LLM, then synthesizes speech serially (ASR → LLM → TTS) stacks three latencies and loses paralinguistic signal (tone, hesitation) at the first hop. Native audio models collapse the stack: speech in, speech out, one session — but only an architecture built for continuous bidirectional flow can use them.
Interruption is the defining feature, not an edge case. Measurements of real conversations show humans overlap and barge in constantly; an agent that finishes its paragraph regardless reads as a machine immediately. Handling it correctly is a systems problem that spans the whole loop: detect user speech during playback (echo cancellation so the agent doesn't trigger on itself), signal the model to stop generating, cancel the audio already queued downstream (client buffers hold seconds of audio — flushing them is your job, not the model's), and record in the session what the user actually heard versus what was generated, or the transcript lies to every future turn.
State is different too. A live session accumulates context inside the model connection itself; connections have hard duration limits and die on network changes. Without resumption architecture, every drop is a lobotomy — the agent forgets mid-task. And cost is per-minute, not per-token: an always-open audio session bills continuously, so idle detection, session timeouts, and knowing when to fall back to turn-based text are economic architecture. Teams that treat streaming as 'the same agent, different transport' rediscover each of these as a production incident; the point of studying the architecture is to design for them on day one.
The architecture: every piece explained
Top row: the transport and the loop. The client captures microphone audio (16kHz PCM chunks, typically 20–100ms frames) and optionally camera/screen frames, shipped over WebSocket (or WebRTC for lower jitter) to your agent server. There, run_live replaces the turn-based entry point: it takes a LiveRequestQueue — the upstream pipe your transport handler pushes frames into (send_realtime for audio bytes, send_content for text/control) — and returns an async stream of events. Two concurrent pumps result: one task drains the client socket into the queue; another drains ADK events back to the socket. The live model session is the persistent connection to the Gemini Live API: it runs its own VAD on the incoming audio, decides when the user has yielded the floor, and generates native audio responses (or text for your own TTS) token-by-token.
Middle row: the conversational machinery. Streaming events carry partial content — text deltas flagged partial, audio chunks, and transcriptions of both user speech and model speech, so your UI can caption live and your session log stays textual. Interruption arrives as a distinct signal when the model's VAD detects user speech over playback: ADK marks the in-flight response interrupted, the model halts generation, and your transport layer must flush queued audio on the client immediately — the event tells you what was cut so the session records reality. Tool calls mid-stream work like turn-based ADK — function declarations, calls, responses — but asynchronously relative to speech: the model can acknowledge ('let me check that') while your async tool executes, then weave the result into its next utterance; slow tools should report progress or the conversation dies of silence.
Bottom rows: continuity and richness. Session resumption: the live connection issues resumption handles; on disconnect (network change, duration limit) you reconnect with the handle and the model restores conversational context — your server keeps the ADK session (events, state) as the durable record regardless. Streaming tools invert the direction: a tool can yield a continuous feed (stock ticks, a video analysis) that the model monitors and reacts to, enabling 'tell me when the build finishes'. The ops strip is the engineering that makes it shippable: an end-to-end latency budget with per-hop measurements, connection lifecycle management (idle timeouts, max duration, graceful handoff), and per-minute cost telemetry.
End-to-end flow
Walk a support call. The browser opens a WebSocket to your FastAPI/ADK server; the handler creates a session, a LiveRequestQueue, and starts run_live with the support agent (tools: lookup_order, issue_refund). Two tasks spin up: socket→queue and events→socket. The user speaks: 'Hi, my order arrived damaged.' Audio frames flow up; the Live API's VAD detects end of speech; within ~300ms audio tokens flow back — 'Sorry to hear that — can I get your order number?' — played as they arrive. Input and output transcriptions ride alongside as events; your UI captions both; the ADK session logs them as the textual record.
The user gives the number; the model emits a function call for lookup_order mid-stream while saying 'One moment.' Your async tool hits the order service (200ms); the function response enters the live session; the model resumes with the details woven in: 'I see it — the ceramic lamp, delivered Tuesday…'. The user barges in over that sentence: 'Yes, that one, I just want a refund.' The model's VAD catches speech-over-playback; an interruption event fires; your client-side handler flushes ~2 seconds of buffered audio instantly (the agent falls silent mid-word — the correct behavior); the session marks the agent's utterance as cut at the interruption point. The model processes the new speech and responds to the refund request — a before_tool callback gates issue_refund above threshold exactly as in turn-based ADK; policy code is transport-agnostic.
Eight minutes in, the user walks from wifi to cellular; the WebSocket dies. The client reconnects with the session id; your server re-establishes the live connection using the stored resumption handle; the model still 'remembers' the lamp and the pending refund — the user hears a beat of silence, not amnesia. At minute twelve the conversation ends; idle detection closes the live session after 30 seconds of silence (per-minute billing stops), while the ADK session persists — the refund approval workflow later resumes it in text mode, because nothing about the session ties it to voice. One conversation, two transports, one event-sourced record.