Why it matters

Prefix tuning is early PEFT method. Understanding shapes evolution.

Advertisement

The architecture

Learn task-specific vectors prepended to K, V per layer.

Frozen base.

Small extra params.

Prefix tuning structureTask prefixeslearned vectorsPrepend to K, Vper layerFrozen baseno other updatesLi + Liang 2021; competes with FT on generation tasks
Prefix tuning.
Advertisement

How it works end to end

Li + Liang 2021.

Effective for generation.

Sensitive to init.

Prompt tuning simplifies.