In recent times, Large Language Models (LLMs) have gained popularity for his or her ability to answer user queries in a more human-like manner, completed through reinforcement learning. Nevertheless, aligning these LLMs with human preferences in reinforcement learning from human feedback (RLHF) can result in a phenomenon often known as reward hacking. This happens when LLMs exploit flaws within the reward model (RM), achieving high rewards without fulfilling the underlying objectives, as illustrated in Figure 1(b). Reward hacking raises concerns equivalent to degraded performance, checkpoint selection challenges, potential biases, and, most critically, safety risks.

The first challenges identified in designing RMs to mitigate reward hacking include distribution shifts and inconsistent preferences within the preference dataset. Distribution shifts arise resulting from policy drift during RL, resulting in a deviation from the offline preference dataset. Inconsistent preferences stem from noisy binary labels, introducing low inter-labeler agreement and impacting RM robustness. To deal with these challenges, existing approaches have explored strategies like KL regularization, energetic learning, and prediction ensembling (ENS). Nevertheless, these methods face efficiency issues, reliability concerns, and struggle with preference inconsistencies.

To tackle these challenges, this paper proposes Weight Averaged Reward Models (WARM) (illustrated in Figure 1(a)), an easy, efficient, and scalable strategy for obtaining a reliable and robust RM. WARM combines multiple RMs through linear interpolation in the load space, providing advantages equivalent to efficiency, improved reliability under distribution shifts, and enhanced robustness to label corruption. The variety across fine-tuned weights is a key contributor to the effectiveness of WARM.

WARM is in comparison with prediction ensembling (ENS), showcasing its efficiency and practicality by requiring a single model at inference time, eliminating memory and inference overheads. Empirical results indicate that WARM performs similarly to ENS when it comes to variance reduction but exhibits superiority under distribution shifts. The paper introduces the concept of linear mode connectivity (LMC) as a key consider WARM’s success, demonstrating its ability to memorize less and generalize higher than ensembling predictions. There are 3 observations which are made within the experiments and are empirically proven in Figure 3 and 4:

• Commentary 1 (LMC): The accuracy of the interpolated model is no less than pretty much as good because the interpolation of the person accuracies.
• Commentary 2 (WA and ENS): Weight averaging and prediction ensembling perform similarly.
• Commentary 3 (WA and ENS): The accuracy gains of WA over ENS grow as data moves away from the training distribution. 

The advantages of WARM extend beyond its primary goals. It aligns with the updatable machine learning paradigm, allowing parallelization in federated learning scenarios. WARM could contribute to privacy and bias mitigation by reducing memorization of personal preferences. The tactic shows potential for combining RMs trained on different datasets, supporting iterative and evolving preferences. Further exploration includes extending WARM to direct preference optimization strategies.

Despite its innovation, WARM has limitations in comparison with prediction ensembling methods, including potential limitations in handling diverse architectures and uncertainty estimation. WARM doesn’t entirely eliminate spurious correlations or biases in preference data, suggesting the necessity for added methods for a comprehensive solution. Lastly, WARM focuses on enhancing reward modeling and needs to be considered inside the broader context of responsible AI to deal with safety risks from misalignment.

In conclusion, Weight Averaged Reward Models (WARM) offer a promising solution to challenges in reward modeling, enhancing alignment in RLHF. The paper’s empirical results and theoretical insights position WARM as a worthwhile contribution toward creating more aligned, transparent, and effective AI systems.

Take a look at the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you happen to like our work, you’ll love our newsletter..

Don’t Forget to affix our Telegram Channel