The recent developments in the sphere of Artificial Intelligence, especially the introduction of Large Language Models, have paved the best way for AI in almost every domain. Foundation models, similar to ChatGPT and Stable Diffusion, have remarkable generalization potential. Nevertheless, training these models from scratch is a challenge due to growing variety of parameters.

The approach of fine-tuning models is straightforward because it doesn’t involve any additional inference delay. Nevertheless, the relational information of weight matrices is difficult to optimally maintain by conventional fine-tuning techniques, which have a low learning rate. Researchers have been studying the Orthogonal Superb-tuning (OFT) technique, which maintains pairwise angles between neurons during fine-tuning by transforming neurons in the identical layer using the identical orthogonal matrix. Though this method has good potential, the identical limitation arises, which is the large variety of trainable parameters that arise from the high dimensionality of orthogonal matrices.

To beat this challenge, a team of researchers has introduced Orthogonal Butterfly (BOFT), a novel and latest method that addresses parameter efficiency in Orthogonal Superb-tuning. Inspired by the butterfly structures within the Cooley-Tukey fast Fourier transform technique, BOFT produces a dense orthogonal matrix by assembling it with quite a few factorized sparse matrices. With a purpose to express the orthogonal matrix as a product of sparse matrices, computation time should be traded for space.

The team has shared that this method will be understood by comparing it to an information transmission problem on a grid-structured graph, which makes it possible to make use of a wide range of sparse matrix factorization techniques that preserve expressiveness while limiting trainable parameters. BOFT has been inspired by the butterfly graph of the Cooley-Tukey method, with its primary innovation being the butterfly factorization process.  

With the usage of this factorization, a dense matrix with a product of O(log d) sparse matrices, each with O(d) non-zero elements, will be created. BOFT can deliver efficient orthogonal parameterization with only O(d log d) parameters, a substantial reduction from the unique OFT parameterization, by guaranteeing orthogonality for every sparse matrix. BOFT offers a general orthogonal fine-tuning framework and subsumes OFT.

The team has compared BOFT with the block-diagonal structure in OFT, and it has shown that with a view to lower the effective trainable parameters, BOFT and OFT each add sparsity to orthogonal matrices. But for downstream applications, a smaller hypothesis class throughout the orthogonal group has been provided by BOFT’s butterfly structure, which allows for a smoother interpolation between full orthogonal group matrices and identity matrices. With a purpose to emphasize that each low-rank and sparse matrices are families of structured matrices that achieve parameter efficiency, this structured approach has been compared with the low-rank structure in LoRA.

The researchers have summarized their primary contributions as follows.

1. The issues of parameter efficiency in orthogonal fine-tuning have been studied to enhance big models’ adaptability for downstream tasks.

2. A brand new framework has been introduced for information transmission that reframes the challenge of constructing a parameter-efficient dense orthogonal matrix as a problem inside a grid-structured graph.

3. Orthogonal Butterfly (BOFT), a parameter-efficient orthogonal fine-tuning method, has been introduced.

4. Matrix factorization and theoretical explanations for why BOFT considerably lowers trainable parameters while preserving expressivity and training stability have been discussed.

5. BOFT has outperformed the state-of-the-art techniques in adaption applications, demonstrating its superior parameter efficiency and generalization abilities.

Take a look at the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to hitch our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.

Should you like our work, you’ll love our newsletter..