Why it matters

DP-SGD is workhorse of private training. Understanding shapes implementation.

Advertisement

The architecture

Per-example gradient clipping.

Add Gaussian noise scaled by clip.

Poisson subsampling per batch.

DP-SGD stepsPer-example gradsclip normNoiseGaussian scaledSubsamplePoisson selectionMoments accountant tracks tight privacy loss over many steps
DP-SGD.
Advertisement

How it works end to end

Clip: L2 norm to C.

Noise: Gaussian with std = sigma * C.

Poisson sampling: each ex included with prob q.

Moments accountant: track epsilon.