In machine learning, generative models that may produce images based on text inputs have made significant progress lately, with various approaches showing promising results. While these models have attracted considerable attention and potential applications, aligning them with human preferences stays a primary challenge on account of differences between pre-training and user-prompt distributions, leading to known issues with the generated images.

Several challenges arise when generating images from text prompts. These include difficulties with accurately aligning text and pictures, accurately depicting the human body, adhering to human aesthetic preferences, and avoiding potential toxicity and biases within the generated content. Addressing these challenges requires greater than just improving model architecture and pre-training data. One approach explored in natural language processing is reinforcement learning from human feedback, where a reward model is created through expert-annotated comparisons to guide the model toward human preferences and values. Nevertheless, this annotation process can take effort and time.

To take care of those challenges, a research team from China has presented a novel solution to generating images from text prompts. They introduce ImageReward, the primary general-purpose text-to-image human preference reward model, trained on 137k pairs of expert comparisons based on real-world user prompts and model outputs.

To construct ImageReward, the authors used a graph-based algorithm to pick out various prompts and provided annotators with a system consisting of prompt annotation, text-image rating, and image rating. Additionally they recruited annotators with not less than college-level education to make sure a consensus within the rankings and rankings of generated images. The authors analyzed the performance of a text-to-image model on various kinds of prompts. They collected a dataset of 8,878 useful prompts and scored the generated images based on three dimensions. Additionally they identified common problems in generated images and located that body problems and repeated generation were probably the most severe. They studied the influence of “function” words in prompts on the model’s performance and located that proper function phrases improve text-image alignment.

The experimental step involved training ImageReward, a preference model for generated images, using annotations to model human preferences. BLIP was used because the backbone, and a few transformer layers were frozen to forestall overfitting. Optimal hyperparameters were determined through a grid search using a validation set. The loss function was formulated based on the ranked images for every prompt, and the goal was to robotically select images that humans prefer.

Within the experiment step, the model is trained on a dataset of over 136,000 pairs of image comparisons and is compared with other models using preference accuracy, recall, and filter scores. ImageReward outperforms other models, with a preference accuracy of 65.14%. The paper also includes an agreement evaluation between annotators, researchers, annotator ensemble, and models. The model is shown to perform higher than other models by way of image fidelity, which is more complex than aesthetics, and it maximizes the difference between superior and inferior images. As well as, an ablation study was conducted to research the impact of removing specific components or features from the proposed ImageReward model. The major results of the ablation study is that removing any of the three branches, including the transformer backbone, the image encoder, and the text encoder, would result in a big drop within the preference accuracy of the model. Particularly, removing the transformer backbone would cause probably the most significant performance drop, indicating the critical role of the transformer within the model.

In this text, we presented a brand new investigation made by a Chinese team that introduced ImageReward. This general-purpose text-to-image human preference reward model addresses issues in generative models by aligning with human values. They created a pipeline for annotation and a dataset of 137k comparisons and eight,878 prompts. Experiments showed ImageReward outperformed existing methods and could possibly be a great evaluation metric. The team analyzed human assessments and planned to refine the annotation process, extend the model to cover more categories and explore reinforcement learning to push text-to-image synthesis boundaries.

Try the Paper and Github. Don’t forget to affix our 20k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you might have any questions regarding the above article or if we missed anything, be happy to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club