Text-to-image models have been the cornerstone of each AI discussion for the last 12 months. The advancement in the sphere happened quite rapidly, and because of this, we’ve impressive text-to-image models. Generative AI has entered a brand new phase.

Diffusion models were the important thing contributors to this advancement. They’ve emerged as a robust class of generative models. These models are designed to generate high-quality images by slowly denoising the input right into a desired image. Diffusion models can capture hidden data patterns and generate diverse and realistic samples.

The rapid advancement of diffusion-based generative models has revolutionized text-to-image generation methods. You possibly can ask for a picture, whatever you’ll be able to consider, describe it, and the models can generate it for you quite accurately. As they progress further, it’s getting obscure which images are generated by AI. 

Nonetheless, there may be a difficulty here. These models solely depend on textual descriptions to generate images. You possibly can only “describe” what you wish to see. Furthermore, they usually are not easy to personalize as that might require fine-tuning usually. 

Imagine doing an interior design of your own home, and you’re employed with an architect. The architect could only give you designs he did for previous clients, and once you attempt to personalize some a part of the design, he simply ignores it and offers you one other used style. Doesn’t sound very pleasing, does it? This is likely to be the experience you’ll get with text-to-image models in the event you are searching for personalization.

Thankfully, there have been attempts to beat these limitations. Researchers have explored integrating textual descriptions with reference images to attain more personalized image generation. While some methods require fine-tuning on specific reference images, others retrain the bottom models on personalized datasets, resulting in potential drawbacks in fidelity and generalization. Moreover, most existing algorithms cater to specific domains, leaving gaps in handling multi-concept generation, test-time fine-tuning, and open-domain zero-shot capability.

So, today we meet with a brand new approach that brings us closer to open-domain personalization—time to satisfy with Subject-Diffusion.

Subject-Diffusion is an modern open-domain personalized text-to-image generation framework. It utilizes just one reference image and eliminates the necessity for test-time fine-tuning. To construct a large-scale dataset for personalized image generation, it builds upon an automatic data labeling tool, leading to the Subject-Diffusion Dataset (SDD) with a powerful 76 million images and 222 million entities.

Subject-Diffusion has three major components: location control, fine-grained reference image control, and a focus control. Location control involves adding mask images of major subjects through the noise injection process. Fantastic-grained reference image control uses a combined text-image information module to enhance the mixing of each granularities. To enable the sleek generation of multiple subjects, attention control is introduced during training.

Subject-Diffusion achieves impressive fidelity and generalization, able to generating single, multiple, and human-subject personalized images with modifications to shape, pose, background, and elegance based on only one reference image per subject. The model also enables smooth interpolation between customized images and text descriptions through a specially designed denoising process. Quantitative comparisons show that Subject-Diffusion outperforms or matches other state-of-the-art methods, each with and without test-time fine-tuning, on various benchmark datasets.

Take a look at the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 27k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.