Why it matters

IA3 is simple efficient PEFT. Understanding shapes options.

Advertisement

The architecture

Learned vectors multiply key + value + FFN activations.

Very few added parameters.

IA3 structureLearned vectorsper-dim scalingMultiply activationsK + V + FFNFew extra paramsefficientLiu et al. 2022 IA3; competitive with LoRA at fewer params
IA3 theory.
Advertisement

How it works end to end

IA3 = Infused Adapter by Inhibiting and Amplifying.

PEFT library supports.

Competitive with LoRA.