Why it matters
Top-p is standard sampling. Understanding shapes decoding.
Advertisement
The architecture
Sort tokens desc.
Take smallest prefix summing to >= p.
Renormalize + sample.
Advertisement
How it works end to end
Common p: 0.9 - 0.95.
Renormalize truncated distribution.
Combined with temperature.