Large Language Models have made an indelible mark on the Artificial Intelligence community. Models like GPT, T5, PaLM, etc., are exponentially becoming popular. These models imitate humans by learning to read, summarize and generate textual data. Their recent impact on AI has helped contribute to a big selection of industries like healthcare, finance, education, entertainment, etc.

Aligning Large Language Models to human values and intentions has been a relentless challenge in the sector of Generative AI, specifically when it comes to being comprehensive, respectful, and compliant. With the immense popularity of GPT-based ChatGPT, this issue has come into the limelight. Current AI systems heavily rely on supervised fine-tuning with human instructions and annotations and reinforcement learning from human feedback (RLHF) to align the models with human preferences. Nevertheless, this approach requires extensive human supervision, which is each expensive and potentially problematic. This results in issues in quality, reliability, diversity, and undesirable biases present in human-provided annotations.

To handle these issues and minimize the dependence of LLMs on intensive human annotations, a team of researchers proposed an approach called SELF-ALIGN. SELF-ALIGN has been introduced to process the aligning of LLM-based AI agents with human values, and that too virtually and annotation-free. It utilizes a small set of human-defined principles or rules to guide the behavior of the AI agents when generating responses to user queries.

The researchers have applied the SELF-ALIGN approach to the LLaMA-65b base language model. An AI assistant named Dromedary has been developed, which achieves significant performance improvements in comparison with the present AI systems, including Text-Davinci-003 and Alpaca, using fewer than 300 lines of human annotations. The code, LoRA weights of Dromedary, and the synthetic training data have been open-sourced to encourage further research in aligning LLM-based AI agents with enhanced supervision efficiency, reduced biases, and improved controllability.

The approach involves 4 stages –

1.   Self-Instruct: This stage employs the self-instruct mechanism by generating synthetic instructions using 175 seed prompts and an extra 20 topic-specific prompts. The aim of those instructions is to offer a comprehensive range of contexts and scenarios for the AI system to learn from.

2.   Principle-Driven Self-Alignment: On this stage, a small set of 16 human-written principles is provided in English, outlining the desirable quality of the system-produced responses. These principles function guidelines for generating helpful, ethical, and reliable responses. The approach utilizes in-context learning (ICL) with a number of demonstrations as an instance how the AI system adheres to the principles when formulating responses in several cases.

3.   Principle Engraving: On this stage, the unique LLM is fine-tuned using the self-aligned responses generated by the LLM through prompting. In the course of the fine-tuning process, the principles and demonstrations are pruned. This fine-tuned LLM can directly generate responses that align well with the principles. 

4.   Verbose Cloning: The ultimate stage involves using context distillation to boost the system’s ability to supply more comprehensive and elaborate responses. This system enables the system to generate detailed and thorough responses.

In conclusion, Dromedary, the bootstrap LLM, seems promising to greatly align itself with minimal human supervision.

Take a look at the Paper and Github link. Don’t forget to hitch our 21k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you have got any questions regarding the above article or if we missed anything, be happy to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club