Why it matters

Audio in LLMs enables speech + audio understanding. Understanding shapes design.

Advertisement

The architecture

Whisper: encoder-decoder ASR.

HuBERT / Wav2Vec 2.0: SSL features.

Audio encodersAudio waveformraw or melWhisper encoderor wav2vecFeaturesfor LLMQwen-Audio, GPT-4o use audio encoders + LLM; growing area
Audio encoders.
Advertisement

How it works end to end

Preprocessing: waveform to mel spectrogram.

Encoder features → LLM.

Real-time streaming variants.