Natural language processing is one area where AI systems are making rapid strides, and it is necessary that the models have to be rigorously tested and guided toward safer behavior to scale back deployment risks. Prior evaluation metrics for such sophisticated systems focused on measuring language comprehension or reasoning in vacuums. But now, models are being taught for actual, interactive work. Which means that benchmarks need to judge how models perform in social settings.

Interactive agents will be put through their paces in text-based games. Agents need planning abilities and the power to know the natural language to progress in these games. Agents’ immoral tendencies ought to be considered alongside their technical talents while setting benchmarks.

A brand new work by the University of California, Center For AI Safety, Carnegie Mellon University, and Yale University proposes the Measuring Agents’ Competence & Harmfulness In A Vast Environment of Long-horizon Language Interactions (MACHIAVELLI) benchmark. MACHIAVELLI is an advancement in evaluating an agent’s capability for planning in naturalistic social settings. The setting is inspired by text-based Select Your Own Adventure games available at choiceofgames.com, which actual humans developed. These games feature high-level decisions while giving agents realistic objectives while abstracting away low-level environment interactions.

The environment reports the degree to which agent acts are dishonest, lower utility, and seek power, amongst other behavioral qualities, to maintain tabs on unethical behavior. The team achieves this by following the below-mentioned steps:

1. Operationalizing these behaviors as mathematical formulas
2. Densely annotating social notions within the games, corresponding to characters’ wellbeing
3. Using the annotations and formulas to supply a numerical rating for every behavior. 

They display empirically that GPT-4 (OpenAI, 2023) is simpler at collecting annotations than human annotators.

Artificial intelligence agents face the identical internal conflict as humans do. Like language models trained for next-token prediction often produce toxic text, artificial agents trained for goal optimization often exhibit immoral and power-seeking behaviors. Amorally trained agents may develop Machiavellian strategies for maximizing their rewards on the expense of others and the environment. By encouraging agents to act morally, this trade-off will be improved.

The team discovers that moral training (nudging the agent to be more ethical) decreases the incidence of harmful activity for language-model agents. Moreover, behavioral regularization restricts undesirable behavior in each agents without substantially decreasing reward. This work contributes to the event of trustworthy sequential decision-makers.

The researchers try techniques like a man-made conscience and ethics prompts to regulate agents. Agents will be guided to display less Machiavellian behavior, although much progress stays possible. They advocate for more research into these trade-offs and emphasize expanding the Pareto frontier somewhat than chasing after limited rewards.

Try the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to affix our 18k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

🚀 Check Out 100’s AI Tools in AI Tools Club