Diffusion models are a set of generative models that work by adding noise to the training data after which learn to get better the identical by reversing the noising process. This process allows these models to realize state-of-the-art image quality, making them one of the vital significant developments in Machine Learning (ML) up to now few years. Their performance, nonetheless, is greatly determined by the distribution of the training data (mainly web-scale text-image pairs), which results in issues like human aesthetic mismatch, biases, and stereotypes.

Previous works give attention to using curated datasets or intervening within the sampling process to handle the abovementioned issues and achieve controllability. Nonetheless, these methods affect the sampling time of the model without improving its inherent capabilities. On this work, researchers from Pinterest have proposed a reinforcement learning (RL) framework for fine-tuning diffusion models to realize results which might be more aligned with human preferences.

The proposed framework enables training over thousands and thousands of prompts across diverse tasks. Furthermore, to make sure that the model generates diverse outputs, the researchers used a distribution-based reward function for reinforcement learning fine-tuning. Moreover, the researchers also performed multi-task joint training in order that the model is healthier equipped to cope with a various set of objectives concurrently.

For evaluation, the authors considered three separate reward functions – image composition, human preference, and variety and fairness. They used the ImageReward model to calculate the human preference rating, which was then used because the reward throughout the model’s training. In addition they compared their framework with various baseline models akin to ReFL, RAFT, DRaFT, etc.

• They found that their method is generalizable to all of the rewards and got the very best rank by way of human preference. They hypothesized that the ReFL model is influenced by the reward hacking problem (the model over-optimizes a single metric at the fee of overall performance). In contrast, their method is rather more robust to those effects.
• The outcomes show that the SDv2 model is biased towards light skin tone for images of dentists and judges, whereas their method has a rather more balanced distribution.
• The proposed framework can be capable of tackle the issue of compositionality in diffusion models, i.e., generating different compositions of objects in a scene, and performs a lot better than the SDv2 model.
• Lastly, by way of multi-reward joint optimization, the model outperforms the bottom models on all three tasks.

In conclusion, to handle the problems with the prevailing diffusion models, the authors of this research paper have introduced a scalable RL training framework that fine-tunes diffusion models to realize higher results. The strategy performed significantly higher than existing models and demonstrated its superiority in generality, robustness, and the flexibility to generate diverse images. With this work, the authors aim to encourage future research on this topic to further enhance diffusion models’ capabilities and mitigate significant issues like bias and fairness.

Try the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

For those who like our work, you’ll love our newsletter..

Don’t Forget to hitch our Telegram Channel