A recent study has highlighted the effectiveness of natural language feedback in improving the performance of language models. A team of researchers from KAIST has introduced a brand new SelFee model designed explicitly for self-feedback and self-revision generation. Unlike previous approaches, SelFee doesn’t require external, significant language or task-specific models to generate high-quality responses.

SelFee is a fine-tuned LLaMA-based instruction-following model that repeatedly revises its answers until it achieves a high-quality response inside a single inference. Based on the given instruction, the model generates an initial solution and self-feedback sequences. By analyzing the content of the generated feedback, the model determines if a revision is required. If that’s the case, it generates a revised answer based on the feedback. This iterative revision process is accomplished inside a single inference, leading to improved solutions in comparison with existing LLaMA-based models.

The researchers collected diverse instruction data from various sources, similar to ShareGPT, Alpaca, Math, Code, and Flan Collection. To deal with the scarcity of feedback and revision data, they augmented the dataset using a distillation process from a teacher model called ChatGPT. This approach allowed them to generate more instances of feedback and revision at a cheaper cost.

To coach the model, the researchers utilized data augmentation techniques using OpenAI API calls. They collected instructions from multiple sources and input them into ChatGPT to generate corresponding answers. They then obtained feedback on the generated answers by querying ChatGPT again. If a revision was deemed essential, ChatGPT revised the reply based on self-generated feedback. This process was repeated until no further modifications were required.

SelFee was trained using the FastChat framework. Based on the instruction, the model was fine-tuned to generate the reply and feedback chain, including revisions. The researchers observed that increasing the minimum required revisions in the course of the inference process improved answer quality. They found that a minimum of three revisions yielded the perfect performance, and even a 7B SelFee model that generated at the very least three revisions outperformed a 13B SelFee model that didn’t require modifications.

When it comes to evaluation, the researchers adopted the Vicuna evaluation setting, which involved 80 diverse queries. As an alternative of conducting a human evaluation, they performed a pilot evaluation using GPT-4 because the evaluator. The relative scores in comparison with ChatGPT were reported, considering the positional bias of GPT-4.

While SelFee demonstrated comparable performance to ChatGPT within the Vicuna evaluation setting, it was found to lack knowledge in areas similar to math, reasoning, factuality, and coding in comparison with ChatGPT.

Overall, SelFee introduces a novel approach to self-feedback and self-revision generation in language models. By fine-tuning the model to revise its answers repeatedly, SelFee achieves improved performance in comparison with existing models. The research findings highlight the importance of iterative revision in enhancing the standard of language model responses and suggest that increasing the inference computation of a model could also be simpler than simply increasing its size.

Check Out The Project Blog, Demo, and Github link. Don’t forget to affix our 22k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you’ve any questions regarding the above article or if we missed anything, be at liberty to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club