Home Community Stanford and UT Austin Researchers Propose Contrastive Preference Learning (CPL): A Easy Reinforcement Learning RL-Free Method for RLHF that Works with Arbitrary MDPs and off-Policy Data

Stanford and UT Austin Researchers Propose Contrastive Preference Learning (CPL): A Easy Reinforcement Learning RL-Free Method for RLHF that Works with Arbitrary MDPs and off-Policy Data

0
Stanford and UT Austin Researchers Propose Contrastive Preference Learning (CPL): A Easy Reinforcement Learning RL-Free Method for RLHF that Works with Arbitrary MDPs and off-Policy Data

The challenge of matching human preferences to big pretrained models has gained prominence within the study as these models have grown in performance. This alignment becomes particularly difficult when there are unavoidably poor behaviours in greater datasets. For this issue, reinforcement learning from human input, or RLHF has develop into popular. RLHF approaches use human preferences to tell apart between acceptable and bad behaviours to enhance a known policy. This approach has demonstrated encouraging outcomes when used to regulate robot rules, enhance image generation models, and fine-tune large language models (LLMs) using less-than-ideal data. There are two stages to this procedure for nearly all of RLHF algorithms. 

First, user preference data is gathered to coach a reward model. An off-the-shelf reinforcement learning (RL) algorithm optimizes that reward model. Regretfully, there must be a correction in the muse of this two-phase paradigm. Human preferences have to be allocated by the discounted total of rewards or partial return of every behaviour segment for algorithms to develop reward models from preference data. Recent research, nonetheless, challenges this theory, suggesting that human preferences must be based on the regret of every motion under the perfect policy of the expert’s reward function. Human evaluation might be intuitively focused on optimality slightly than whether situations and behaviours provide greater rewards. 

Due to this fact, the optimal advantage function, or the negated regret, will be the ideal number to learn from feedback slightly than the reward. Two-phase RLHF algorithms use RL of their second phase to optimize the reward function known in the primary phase. In real-world applications, temporal credit project presents a wide range of optimization difficulties for RL algorithms, including the instability of approximation dynamic programming and the high variance of policy gradients. Consequently, earlier works restrict their reach to avoid these problems. For instance, contextual bandit formulation is assumed by RLHF approaches for LLMs, where the policy is given a single reward value in response to a user query. 

The only-step bandit assumption is broken because user interactions with LLMs are multi-step and sequential, even while this lessens the requirement for long-horizon credit project and, in consequence, the high variation of policy gradients. One other example is the applying of RLHF to low-dimensional state-based robotics issues, which works well for approximation dynamic programming. Nevertheless, it has yet to be scaled to higher-dimensional continuous control domains with picture inputs, that are more realistic. Generally, RLHF approaches require reducing the optimisation constraints of RL by making restricted assumptions concerning the sequential nature of problems or dimensionality. They typically mistakenly imagine that the reward function alone determines human preferences.

In contrast to the widely used partial return model, which considers the overall rewards, researchers from Stanford University, UMass Amherst and UT Austin provide a novel family of RLHF algorithms on this study that employs a regret-based model of preferences. In contrast to the partial return model, the regret-based approach gives precise information on the perfect plan of action. Fortunately, this removes the need for RL, enabling us to tackle RLHF issues with high-dimensional state and motion spaces within the generic MDP framework. Their fundamental finding is to create a bijection between advantage functions and policies by combining the regret-based preference framework with the Maximum Entropy (MaxEnt) principle. 

They will establish a purely supervised learning objective whose optimum is the perfect policy under the expert’s reward by trading optimization over benefits for optimization over policies. Because their method resembles well known contrastive learning objectives, they call it Contrastive Preference Learning—three principal advantages of CPL over earlier efforts. First, because CPL matches the optimal advantage exclusively using supervised goals—slightly than using dynamic programming or policy gradients—it will possibly scale in addition to supervised learning. Second, CPL is totally off-policy, making using any offline, less-than-ideal data source possible. Lastly, CPL enables preference searches over sequential data for learning on arbitrary Markov Decision Processes (MDPs). 

So far as they know, previous techniques for RLHF have yet to satisfy all three of those requirements concurrently. They illustrate CPL’s performance on sequential decision-making issues using sub-optimal and high-dimensional off-policy inputs to prove that it adheres to the abovementioned three tenets. Interestingly, they display that CPL may learn temporally prolonged manipulation rules within the MetaWorld Benchmark by efficiently utilising the identical RLHF fine-tuning process as dialogue models. To be more precise, they use supervised learning from high-dimensional picture observations to pre-train policies, which they then fine-tune using preferences. CPL can match the performance of earlier RL-based techniques without the necessity for dynamic programming or policy gradients. Additionally it is 4 times more parameter efficient and 1.6 times quicker concurrently. On five tasks out of six, CPL outperforms RL baselines when utilizing denser preference data. Researchers can avoid the need for reinforcement learning (RL) by employing the concept of maximum entropy to create Contrastive Preference Learning (CPL), an algorithm for learning optimum policies from preferences without learning reward functions.


Take a look at the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

If you happen to like our work, you’ll love our newsletter..

We’re also on Telegram and WhatsApp.


Aneesh Tickoo is a consulting intern at MarktechPost. He’s currently pursuing his undergraduate degree in Data Science and Artificial Intelligence from the Indian Institute of Technology(IIT), Bhilai. He spends most of his time working on projects aimed toward harnessing the facility of machine learning. His research interest is image processing and is obsessed with constructing solutions around it. He loves to attach with people and collaborate on interesting projects.


🔥 Meet Retouch4me: A Family of Artificial Intelligence-Powered Plug-Ins for Photography Retouching

LEAVE A REPLY

Please enter your comment!
Please enter your name here