Why it matters

Dynamic NTK adapts. Understanding shapes advanced extension.

Advertisement

The architecture

Per-sequence: compute scale from actual length.

Larger scale for longer seq.

Dynamic NTK flowActual seq lengthat inferenceCompute scaleadaptiveApply NTK scalingwith computed scaleZero fine-tune option; quality varies by length; useful when length unknown at train
Dynamic NTK.
Advertisement

How it works end to end

No fine-tune version.

Compute scale as max(1, len / trained_len).

Applied at attention time.