Why it matters

Multimodal unlocks new applications. Understanding shapes modern agent capabilities.

Advertisement

The architecture

Content parts: text + image + audio + video.

Gemini accepts multipart messages natively.

Multimodal inputText parttraditionalImage partURL or bytesAudio + video partsfor Gemini 2.0+Each modality has size and format constraints; check Gemini docs
Multimodal message.
Advertisement

How it works end to end

Image input: URL or inline base64. Common formats (PNG, JPEG).

Audio: for transcription or understanding.

Video: keyframes analyzed for content understanding.

Output: currently mostly text; some models generate images.