Pretrained language models (LMs) are commonly finetuned to adapt them to recent domains or tasks, a process often called finetuning. While finetuning allows for adaptation to varied functions with small amounts of in-domain data, it could actually be prohibitively expensive for big LMs. 

Parameter-efficient finetuning (PEFT) methods offer an answer by updating only a fraction of the weights, reducing memory usage and training time. Adapters, a typical PEFT approach, learn edits that will be added to a subset of model weights or operate alongside the frozen base model. Recent advancements like LoRA and its variants reduce the variety of trainable parameters through the use of low-rank approximations during adapter training.

Nevertheless, a major aspect of current PEFT methods is their deal with modifying weights fairly than representations, despite prior research indicating that representations encode wealthy semantic information. Representation Finetuning (ReFT) methods have been proposed in response to this by a team of researchers from Stanford and Pr(Ai)2R Group.

As an alternative of adapting model weights, ReFT methods train interventions to control a small fraction of model representations, steering model behaviors to resolve downstream tasks at inference time. Their approach draws inspiration from recent work in LM interpretability, which intervenes on representations to discover causal mechanisms and steer model behaviors at inference time.

One notable instance of the ReFT family is the Low-rank Linear Subspace ReFT (LoReFT), which intervenes on hidden representations within the linear subspace spanned by a low-rank projection matrix. LoReFT builds directly on existing methods like distributed alignment search (DAS), demonstrating state-of-the-art performance on various benchmarks while using significantly fewer parameters than traditional PEFT methods. Their results suggest that ReFT methods offer more efficient and effective alternatives to weight-based PEFTs, deserving further exploration across different model families and domains.

Future research directions for ReFT include exploring its effectiveness on other model families and vision-language models and automating hyperparameter search. Moreover, investigating more practical interventions for specific tasks and exploring the facility of learned orthogonal subspaces are areas of interest. ReFT advances neural network interpretability research and contributes insights back to the sphere, difficult traditional approaches to interpreting individual neurons in isolation.

When it comes to evaluation practices, it’s essential to determine benchmarks that allow for fair comparisons of PEFTs and ReFTs, including compute- or time-matched hyperparameter-tuning comparisons and disallowing tuning or model selection based on the test set to mitigate overfitting and ensure real-world performance assessment.

Take a look at the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

For those who like our work, you’ll love our newsletter..

Don’t Forget to hitch our 40k+ ML SubReddit

Wish to get in front of 1.5 Million AI Audience? Work with us here