The well-known Artificial Intelligence (AI)-based chatbot, i.e., ChatGPT, which has been built on top of GPT’s transformer architecture, uses the strategy of Reinforcement Learning from Human Feedback (RLHF). RLHF is an increasingly necessary method for utilizing the potential of pre-trained Large Language Models (LLMs) to generate more helpful, truthful responses which can be in keeping with human preferences.
In RLHF, a language model is trained to supply responses that maximize the learned reward through reinforcement learning, after which a reward model is trained based on human preferences for particular prompts. Since gathering human rankings is usually simpler than gathering demos for supervised fine-tuning, this approach streamlines the means of collecting data.
Nevertheless, reward hacking is a subtle problem with RLHF, where the policy gets a big reward without meeting the true objectives. This happens in consequence of the reward model’s limited Out-Of-Distribution (OOD) generalization and potential imperfections in representing human preferences. Being a robust LLM, the language model can provide OOD examples to benefit from flaws within the reward model.
The scenario is further complicated by human preference data, which is steadily skewed and inconsistent on account of task complexity and subjectivity, defects in rating standards, and the low caliber of raters. Verbosity is a preferred example of reward hacking, by which models produce more tokens to look more thorough or higher formatted in responses, but there isn’t any real improvement in quality.
As a way to address these issues, recent research from NVIDIA and the University of Maryland has aimed to mitigate reward hacking by examining how RL algorithms and incentive models affect verbosity and performance. The team has presented an evaluation technique to check various training setups and account for biases in model-based evaluations. The technique has provided a comprehensive knowledge of assorted response durations by evaluating performance on the Pareto front of evaluation rating vs. length.
This process is meant to investigate the trade-off between the LLM’s assessment rating and response duration, allowing for a scientific comparison of various training settings. By various the training hyperparameters, it will probably be evaluated how these modifications affect the ratio of verbosity to reply quality.
The study looks at RL hyperparameters and techniques, equivalent to reward clipping and length penalty, to minimize reward hacking on length. The first goal is to remove the spurious length signal from the reward, though various tuning procedures can yield higher outcomes. To perform this, the team has suggested a two-head reward model that separates representations for length from true preferences. The length head is deleted during RL.
The suggested reward disentangling technique, ODIN, has been used with the assistance of which, even with a more costly tuning budget, the policy was in a position to attain a bigger Pareto front than prior results. Proximal Policy Optimisation (PPO) and ReMax each profit from ODIN’s effectiveness, indicating that it will probably be used to reinforce other RL-tuning methods and lessen length hacking.
In conclusion, this method’s experimental results have shown a noteworthy decrease within the reward model’s association with response duration. The derived strategy performs significantly higher when the standard of the knowledge is prioritized over verbosity. This method successfully reduces the issue of response length-related reward hacking, improving the dependability and utility of LLMs trained using the RLHF paradigm.
Take a look at the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 37k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.
For those who like our work, you’ll love our newsletter..
Don’t Forget to affix our Telegram Channel
Tanya Malhotra is a final 12 months undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and significant pondering, together with an ardent interest in acquiring recent skills, leading groups, and managing work in an organized manner.