Math behind this parameter efficient finetuning method

Advantageous-tuning large pre-trained models is computationally difficult, often involving adjustment of tens of millions of parameters. This traditional fine-tuning approach, while effective, demands substantial computational resources and time, posing a bottleneck for adapting these models to specific tasks. LoRA presented an efficient solution to this problem by decomposing the update matrix during finetuing. To review LoRA, allow us to start by first revisiting traditional finetuing.

In traditional fine-tuning, we modify a pre-trained neural network’s weights to adapt to a brand new task. This adjustment involves altering the unique weight matrix ( W ) of the network. The changes made to ( W ) during fine-tuning are collectively represented by ( Δ W ), such that the updated weights could be expressed as ( W + Δ W ).

Now, quite than modifying ( W ) directly, the LoRA approach seeks to decompose ( Δ W ). This decomposition is a vital step in reducing the computational overhead related to fine-tuning large models.

The intrinsic rank hypothesis suggests that significant changes to the neural network could be captured using a lower-dimensional representation. Essentially, it posits that not all elements of ( Δ W ) are equally essential; as an alternative, a smaller subset of those changes can effectively encapsulate the crucial adjustments.

Constructing on this hypothesis, LoRA proposes representing ( Δ W ) because the product of two smaller matrices, ( A ) and ( B ), with a lower rank. The updated weight matrix ( W’ ) thus becomes:

[ W’ = W + BA ]

On this equation, ( W ) stays frozen (i.e., it just isn’t updated during training). The matrices ( B ) and ( A ) are of lower dimensionality, with their product ( BA ) representing a low-rank approximation of ( Δ W ).

By selecting matrices ( A ) and ( B ) to have a lower rank ( r ), the variety of trainable parameters is significantly reduced. For instance, if ( W ) is a ( d x d ) matrix, traditionally, updating ( W ) would involve ( d² ) parameters. Nevertheless, with ( B ) and ( A ) of sizes ( d x r ) and ( r x d ) respectively, the full variety of parameters reduces to ( 2dr ), which is way smaller when ( r << d ).

The reduction within the variety of trainable parameters, as achieved through the Low-Rank Adaptation (LoRA) method, offers several significant advantages, particularly when fine-tuning large-scale neural networks:

1. Reduced Memory Footprint: LoRA decreases memory needs by lowering the variety of parameters to update, aiding within the management of large-scale models.
2. Faster Training and Adaptation: By simplifying computational demands, LoRA accelerates the training and fine-tuning of enormous models for brand new tasks.
3. Feasibility for Smaller Hardware: LoRA’s lower parameter count enables the fine-tuning of considerable models on less powerful hardware, like modest GPUs or CPUs.
4. Scaling to Larger Models: LoRA facilitates the expansion of AI models with out a corresponding increase in computational resources, making the management of growing model sizes more practical.

Within the context of LoRA, the concept of rank plays a pivotal role in determining the efficiency and effectiveness of the difference process. Remarkably, the paper highlights that the rank of the matrices A and B could be astonishingly low, sometimes as little as one.

Although the LoRA paper predominantly showcases experiments throughout the realm of Natural Language Processing (NLP), the underlying approach of low-rank adaptation holds broad applicability and may very well be effectively employed in training various kinds of neural networks across different domains.

LoRA’s approach to decomposing ( Δ W ) right into a product of lower rank matrices effectively balances the necessity to adapt large pre-trained models to recent tasks while maintaining computational efficiency. The intrinsic rank concept is essential to this balance, ensuring that the essence of the model’s learning capability is preserved with significantly fewer parameters.

References:
[1] Hu, Edward J., et al. “Lora: Low-rank adaptation of enormous language models.” arXiv preprint arXiv:2106.09685 (2021).