Reward shaping, which seeks to develop reward functions that more effectively direct an agent towards desirable behaviors, remains to be a long-standing difficulty in reinforcement learning (RL). It’s a time-consuming procedure that requires skill, is likely to be sub-optimal, and is continuously done manually by constructing incentives based on expert intuition and heuristics. Reward shaping could also be addressed via inverse reinforcement learning (IRL) and preference learning. A reward model could be taught using preference-based feedback or human examples. Each approaches still need significant labor or data collecting, and the neural network-based reward models must be more comprehensible and unable to generalize outside the training data’s domains. 

Researchers from The University of Hong Kong, Nanjing University, Carnegie Mellon University, Microsoft Research, and the University of Waterloo introduce the TEXT2REWARD framework for creating wealthy reward code based on goal descriptions. TEXT2REWARD creates dense reward code (Figure 1 center) based on large language models (LLMs), that are based on a condensed, Pythonic description of the environment (Figure 1 left), given an RL objective (for instance, “push the chair to the marked position”). Then, an RL algorithm like PPO or SAC uses dense reward coding to coach a policy (Figure 1 right). In contrast to inverse RL, TEXT2REWARD produces symbolic rewards with good data-free interpretability. The authors’ free-form dense reward code, in contrast to recent work that used LLMs to jot down sparse reward code (the reward is non-zero only when the episode ends) with hand-designed APIs, covers a wider range of tasks and might make use of proven coding frameworks (resembling NumPy operations over point clouds and agent positions). 

Finally, given the sensitivity of RL training and the paradox of language, the RL strategy may fail to realize the aim or achieve it in ways in which weren’t intended. By applying the learned policy in the actual world, getting user input, and adjusting the reward as needed, TEXT2REWARD solves this issue. They carried out systematic studies on two robotics manipulation benchmarks, MANISKILL2, METAWORLD, and two locomotion environments of MUJOCO. Policies trained with their produced reward code achieve equivalent or greater success rates and convergence speeds than the bottom truth reward code meticulously calibrated by human specialists on 13 out of 17 manipulation tasks. 

With a hit rate of over 94%, TEXT2REWARD learns 6 unique locomotor behaviors. Moreover, they show how the simulator-trained strategy could also be applied to a real Franka Panda robot. Their approach may iteratively increase the success rate of learned policy from 0 to over 100% and eliminate task ambiguity with human input in lower than three rounds. In conclusion, the experimental findings showed that TEXT2REWARD could provide interpretable and generalizable dense reward code, enabling a human-in-the-loop pipeline and extensive RL task coverage. They anticipate the outcomes will stimulate more research into the interface between reinforcement learning and code creation.

Try the Paper, Code, and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 31k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

In case you like our work, you’ll love our newsletter..