Why architecture matters here
Neural audio codecs matter because they achieve excellent audio quality at low bitrates (better than classical codecs) AND produce the discrete audio tokens that are the foundation of generative audio models -- making them both a compression advance and the tokenizer for generative audio. Neural codecs have two major values. As compression: they learn a highly efficient audio compression (end-to-end trained) -- achieving better quality at low bitrates than classical hand-engineered codecs (valuable for bandwidth-constrained applications). As a tokenizer for generative audio: they produce discrete audio tokens (a language of audio tokens) -- which is the interface for generative audio models (audio LLMs, TTS, music generation -- which operate on the tokens) -- so neural codecs are foundational to generative audio (the tokenizer). This second value is increasingly important (generative audio -- TTS, music, audio LLMs -- being a major area) -- so neural codecs are foundational to it. For audio compression (low-bitrate quality) and especially generative audio (the audio tokenizer), neural codecs are important, and understanding them (the learned compression, the tokens) is understanding a key audio technology.
The learned-encoder-quantizer-decoder insight is the architectural core, and it's how neural codecs compress and tokenize. The architecture is three learned components. The encoder: a neural network mapping the raw audio waveform to a compact latent representation (learning to extract the essential information into a compact form). The quantizer: turning the continuous latent into discrete tokens (a small set of code indices -- quantizing the latent to a codebook) -- so the representation is discrete and compact (a few tokens). The decoder: a neural network reconstructing the audio waveform from the tokens (learning to regenerate the audio from the compact discrete representation). Trained end-to-end (the encoder, quantizer, and decoder trained together to reconstruct the audio well -- minimizing the reconstruction error -- while the quantizer keeps the representation compact/discrete), the codec learns an efficient compression (the encoder learning what to keep, the decoder learning to reconstruct -- a learned, efficient representation -- often better than hand-engineered). And the discrete tokens (from the quantizer) are the compact representation (for storage/transmission -- compression) and the token interface (for generative models). So the learned encoder-quantizer-decoder (compressing audio to discrete tokens and back, trained end-to-end) is the core of neural codecs (the learned compression producing the tokens). Understanding the learned-encoder-quantizer-decoder core (the learned compression to discrete tokens and back) is understanding how neural codecs work.
And the residual-VQ-bitrate-tradeoff insight is the key mechanism for controlling quality vs bitrate, and it's elegant. The quantizer is typically a residual vector quantizer (RVQ): instead of a single codebook (which would need to be huge for high quality -- impractical), RVQ uses multiple codebooks applied in sequence. The first codebook quantizes the latent (a coarse approximation), the second quantizes the residual (the error left by the first -- refining it), the third quantizes that residual, and so on -- each codebook refining the approximation (quantizing the remaining residual). So the representation is a sequence of tokens (one per codebook) -- and crucially, the number of codebooks controls the bitrate-vs-quality tradeoff: more codebooks (more tokens -- more bits) give higher quality (finer approximation -- the residuals more refined) at a higher bitrate; fewer codebooks (fewer tokens -- fewer bits) give lower bitrate at lower quality (coarser approximation). This lets you trade bitrate for quality by choosing the number of codebooks (a flexible tradeoff -- e.g., few codebooks for a very low bitrate, more for high quality) -- and even do it dynamically (using fewer codebooks when bandwidth is constrained). This RVQ (multiple codebooks refining residuals -- the codebook count trading bitrate for quality) is the elegant key mechanism for the flexible bitrate-quality tradeoff. Understanding the residual-VQ-bitrate-tradeoff (RVQ's multiple codebooks, the count controlling bitrate vs quality) is understanding how neural codecs flexibly trade bitrate for quality.
The architecture: every piece explained
Top row: the idea and components. The idea: learned compression (a neural network learning the audio compression -- versus hand-engineered classical codecs). Encoder: mapping the raw audio waveform to a compact latent representation (learning to extract the essential information). Quantizer (RVQ): turning the continuous latent into discrete tokens (via residual vector quantization -- a small set of code indices). Decoder: reconstructing the audio waveform from the tokens (learning to regenerate the audio).
Middle row: RVQ and tokens. Residual VQ: multiple codebooks applied in sequence (each quantizing the residual left by the previous -- refining the approximation) -- the quantization structure. Bitrate vs quality: the number of codebooks controls the tradeoff (more codebooks -- more tokens/bits -- higher quality; fewer -- lower bitrate, lower quality) -- the flexible tradeoff. Audio tokens: the discrete tokens the codec produces -- the interface for generative audio models (audio LLMs, TTS, music generation -- operating on the tokens -- treating audio like a language). vs classical codecs: neural codecs (learned -- better low-bitrate quality, token interface) vs classical (Opus, AAC -- hand-engineered, mature, low-latency) -- neural's advantages (quality, tokens) and considerations (compute, latency).
Bottom rows: models and constraints. EnCodec / SoundStream: prominent neural codec models (EnCodec from Meta, SoundStream from Google -- the influential neural codecs) -- the models. Latency + streaming: the real-time constraints -- the neural model's compute (encoding/decoding latency) and streaming (processing audio in real-time -- low-latency for interactive use -- e.g., calls) -- considerations for real-time use (versus classical codecs' low latency). The ops strip: bitrate (choosing the bitrate -- the number of RVQ codebooks -- for the quality/bandwidth needs -- and dynamically adapting it), latency (the encoding/decoding latency -- managing it for real-time/streaming use -- the neural compute), and quality eval (evaluating the audio quality -- perceptual quality metrics, listening tests -- confirming the codec's quality at the chosen bitrate).
End-to-end flow
Trace neural codec compression and tokenization. Audio (a waveform) is encoded: the encoder maps it to a compact latent, and the RVQ quantizer turns the latent into discrete tokens (a sequence of code indices -- one per codebook, refining the approximation). The tokens are the compact representation (for storage/transmission -- compression -- far fewer bits than the raw audio) -- or for a generative model (the token interface). To reconstruct, the decoder takes the tokens and regenerates the audio waveform (learning to reconstruct from the compact tokens). So the neural codec compressed the audio to discrete tokens (efficient -- the learned compression) and reconstructed it (the decoder) -- achieving good quality at a low bitrate (the learned efficiency). And the discrete tokens (the audio as a sequence of tokens) are the interface for generative audio (a generative model operating on the tokens -- generating audio token by token). The neural codec (encode to tokens, decode back) provided the compression and the tokenization.
The bitrate-tradeoff and generative vignettes show the two values. A bitrate-tradeoff case: the team needs different quality/bandwidth points -- for a low-bandwidth scenario, they use few RVQ codebooks (a low bitrate -- fewer tokens -- lower quality but fitting the bandwidth); for high quality, more codebooks (a higher bitrate -- more tokens -- higher quality). The RVQ codebook count let them trade bitrate for quality flexibly (per the scenario) -- even dynamically (adapting to available bandwidth). The RVQ tradeoff provided the flexible bitrate. A generative case: the team builds a text-to-speech model that generates audio tokens (the neural codec's tokens) -- the TTS model predicting the sequence of audio tokens (like a language model), and the neural codec's decoder turning the generated tokens into the audio waveform. So the neural codec's tokens are the interface (the generative model operating on the tokens, the codec decoding them to audio) -- foundational to the generative audio. The neural codec was the tokenizer for the generative TTS.
The comparison and latency vignettes complete it. A comparison case: the team compares neural codecs to classical (Opus) -- neural codecs offering better quality at low bitrates (the learned compression) and the token interface (for generative audio) -- but classical codecs having very low latency and maturity (hand-engineered, optimized). For a low-bitrate quality or generative use, they use a neural codec; for a low-latency real-time call where classical suffices, Opus -- matching the codec to the need. A latency case: for a real-time/streaming use (e.g., a live interactive application), the team considers the neural codec's latency (the encoding/decoding compute -- higher than classical) -- ensuring it meets the real-time constraint (a streaming-capable, low-latency neural codec configuration -- or classical if the neural latency is too high) -- managing the latency for the real-time use. The consolidated discipline the team documents: use neural audio codecs for low-bitrate quality (the learned compression -- better than classical at low bitrates) and as the tokenizer for generative audio (the discrete audio tokens -- the interface for audio LLMs, TTS, music generation), understand the encoder-quantizer-decoder architecture (learned compression to discrete tokens and back), use RVQ's codebook count to trade bitrate for quality (flexible -- even dynamic), leverage the audio tokens for generative models, choose neural vs classical codecs by need (neural for low-bitrate quality/generative; classical for low-latency maturity), manage the latency for real-time/streaming use, and evaluate the audio quality -- because neural audio codecs achieve excellent low-bitrate quality (learned compression) and produce the discrete audio tokens that are the foundation of generative audio, both a compression advance and the tokenizer for generative audio.