Why it matters

IB method underlies bottleneck theory. Understanding shapes framework.

Advertisement

The architecture

Objective: minimize I(X; T) - beta * I(T; Y).

Beta controls compression / prediction.

IB objectiveCompressminimize I(X; T)Predictmaximize I(T; Y)Trade-off betacontrols balanceTishby et al. 1999; theoretical framework applied to DL by Tishby + Zaslavsky
Information bottleneck.
Advertisement

How it works end to end

Tishby et al. 1999.

Beta > 1: prediction dominates.

Beta < 1: compression dominates.