Why architecture matters here
Multimodal matters because it collapses brittle multi-step pipelines into native reasoning. The pre-multimodal way to 'understand a receipt' was a pipeline: OCR the image to text (losing layout, failing on poor quality), parse the text with rules or a model (brittle to format variation), then reason — three stages, each a failure point, each losing information the next stage needed. A native multimodal model sees the receipt as an image, understanding layout, handwriting, and structure holistically — the merchant name next to the logo, the total at the bottom, the line items in their columns — reasoning about it directly. This isn't incremental; it's a different capability, turning tasks that required fragile custom pipelines (document understanding, visual question answering, audio comprehension) into a single model call. For any agent whose users interact with the physical or visual world — photos, documents, voice — multimodality is the difference between useful and useless.
The architectural challenge multimodality introduces is media is heavy — in tokens, latency, and cost. A text message is cheap; an image is hundreds to thousands of tokens (depending on resolution and the model's encoding); a document is more; a video is a lot. This changes the economics and the context management: inlining a 20-page PDF into every turn's context is a cost and context-window disaster, so media must live in artifacts (referenced, loaded when needed) rather than perpetually in context; token budgets must account for media weight; and latency is higher (processing an image takes longer than a text token). The context-engineering discipline that matters for text matters more for multimodal — media is the biggest context bloat source, and managing it (artifacts, just-in-time loading, resolution choices) is central to multimodal agent design.
And modality-specific quality is the third reality: a model's text ability doesn't predict its multimodal ability, and multimodal failures are modality-specific. A model might read printed text perfectly but struggle with handwriting, understand clear photos but fail on blurry ones, transcribe clean audio but not noisy audio. So multimodal agents need modality-specific evaluation — a receipt-understanding agent evaluated on real (blurry, varied, handwritten) receipts, not clean examples; an audio agent evaluated on real (accented, noisy, overlapping) speech. The eval discipline from text agents extends, but the eval sets must cover the modality's failure modes, because the ways multimodal agents fail (misreading a smudged digit, missing a quiet word) are specific to the medium and invisible to text evals.
The architecture: every piece explained
Top row: input to grounded answer. Multimodal input arrives as media — images (photos, screenshots), audio (voice notes, recordings), documents (PDFs, scanned forms), video — from users or tools. Content parts is the representation: a message is a list of typed parts (a text part, an image part with the image data or reference, an audio part), so a single message can combine 'what's wrong with this?' (text) and the photo (image part). The multimodal model (Gemini, natively multimodal) processes all parts together — it doesn't OCR-then-reason; it reasons over the image, audio, and text jointly, understanding the media holistically. Grounded reasoning is the output: an answer grounded in the media (what the image shows, what the audio said, what the document contains) — visual question answering, audio comprehension, document extraction.
Middle row: media handling and output. Artifacts keep large media out of the token stream: a big PDF or video is stored as an artifact and referenced, loaded into context only when the agent needs to reason about it (and at appropriate resolution) — the context-management discipline that prevents media from bloating every turn. Tool outputs as media: a tool can return an image (a chart it generated, a screenshot it captured, a diagram) that the agent then reasons about — closing the loop where the agent acts, gets visual feedback, and interprets it. Multimodal output: the agent can generate media — images (via an image-generation tool or model), audio (TTS) — not just consume it. Modality routing: different media may route to different models or paths (a specialized OCR model for dense documents, a general vision model for photos, an audio model for speech) — choosing the right tool per modality.
Bottom rows: applications and economics. Document understanding is a major use case: PDFs, forms, tables, and scanned documents understood natively (layout, structure, handwriting) — extracting fields, answering questions, summarizing — replacing brittle OCR-plus-parsing pipelines. Cost and latency are the constraints: media tokens are expensive (images hundreds-to-thousands of tokens, video far more) and slower to process, so multimodal agents need tighter token budgets and latency awareness than text agents. The ops strip: media handling (artifacts for large media, resolution choices trading cost against detail, just-in-time loading), token budgets (media weight accounted for, context managed against media bloat), and modality-specific evals (eval sets covering the modality's real failure modes — blurry images, noisy audio, varied documents — not clean examples).
End-to-end flow
Trace a multimodal support agent handling a real request. A user photographs a damaged product and sends 'this arrived broken, what are my options?' with the image. The message carries two content parts: the text and the image. The multimodal model reasons over both — seeing the cracked item in the photo, understanding it's damage, connecting it to the text — and responds appropriately (recognizing the damage, offering a refund per policy), grounded in what the image actually shows rather than taking the user's word. No OCR pipeline, no separate image-classification step — the model saw the photo and reasoned about it directly. This is the collapse of a brittle pipeline into native capability.
The document-understanding and media-handling vignettes show the depth. A user uploads a 20-page warranty PDF asking 'does this cover water damage?'. The PDF is stored as an artifact (not inlined into every turn — that would be a token disaster); the agent, needing to answer, loads the relevant pages into context and the model reads them natively (understanding the document's structure, finding the water-damage clause in section 7) — answering with a citation. The artifact discipline kept the 20-page document out of the perpetual context (it's referenced, loaded on demand), and the native document understanding read the PDF's layout and text without a separate OCR-and-parse pipeline. The cost awareness matters: loading all 20 pages at full resolution would be expensive, so the agent loads the relevant pages at appropriate resolution — the media-token economics driving the handling.
The tool-media and eval vignettes complete it. The agent uses a tool that generates a chart of the customer's order history; the tool returns the chart as an image, and the agent reasons about it ('your orders show a pattern of...') — tool output as media, closing the visual loop. And the quality discipline shows in the evals: the team's receipt-processing agent is evaluated not on clean sample receipts but on real ones — blurry phone photos, handwritten totals, faded thermal paper, varied formats — because the model's clean-receipt accuracy didn't predict its real-receipt accuracy, and the failures (misreading a smudged digit, missing a handwritten tip) are modality-specific and invisible to text evals. The consolidated discipline: media as typed content parts to a native multimodal model, large media in artifacts (loaded just-in-time at appropriate resolution), token budgets accounting for media weight, tool outputs as media where visual feedback helps, and modality-specific eval sets covering real-world media failure modes — because multimodality collapses brittle pipelines into native reasoning, but the media's cost, context weight, and modality-specific failures are the new engineering realities.