Text-to-image generation is a term we’re all acquainted with at this point. The era after the stable diffusion release has brought one other intending to image generation, and the advancements afterward made it in order that it is de facto getting difficult to distinguish AI-generated images nowadays. With MidJourney continuously improving and Stability AI releasing updated models, the effectiveness of text-to-image models has reached an especially high level.

Now we have also seen attempts to make these models more personalized. People have worked on developing models that might be used to edit a picture with the assistance of AI, like replacing an object, changing the background, etc., all with a given text prompt. This advanced capability of text-to-image models has also given birth to a cool startup where you’ll be able to generate your personal personalized AI avatars, and it became successful very suddenly. 

Personalized text-to-image generation has been a captivating area of research, aiming to generate latest scenes or forms of a given concept while maintaining the identical identity. This difficult task involves learning from a set of images after which generating latest images with different poses, backgrounds, object locations, dressing, lighting, and styles. While existing approaches have made significant progress, they often depend on test-time fine-tuning, which might be time-consuming and limit scalability. 

Proposed approaches for personalized image synthesis have typically relied on pre-trained text-to-image models. These models are able to generating images but require fine-tuning to learn each latest concept, which necessitates storing model weights per concept. 

What if we could have an alternative choice to this? What if we could have a customized text-to-image generation model that doesn’t depend on test-time fine-tuning in order that we will scale it higher and achieve personalization in a little bit time? Time to satisfy InstantBooth.

To deal with these limitations, InstantBooth proposes a novel architecture that learns the overall concept from input images using a picture encoder. It then maps these images to a compact textual embedding, ensuring generalizability to unseen concepts.

While compact embedding captures the overall idea, it doesn’t include the fine-grained identity details essential to generate accurate images. To tackle this problem, InstantBooth introduces trainable adapter layers inspired by recent advances in language and vision model pre-training. These adapter layers extract wealthy identity information from the input images and inject it into the fixed backbone of the pre-trained model. This ingenious approach successfully preserves the identity details of the input concept while retaining the generation ability and language controllability of the pre-trained model.

Furthermore, InstantBooth eliminates the necessity for paired training data, making it more practical and feasible. As an alternative, the model is trained on text-image pairs without counting on paired images of the identical concept. This training strategy enables the model to generalize well to latest concepts. When presented with images of a brand new concept, the model can generate objects with significant pose and site variations while ensuring satisfactory identity preservation and alignment between language and image.

Overall, InstantBooth has three key contributions to the personalized text-to-image generation problem. First, the test-time finetuning isn’t any longer required. Second, DreamBooth enhances generalizability to unseen concepts by converting input images into textual embeddings. Furthermore, by injecting a wealthy visual feature representation into the pre-trained model, it ensures identity preservation without sacrificing language controllability. Finally, InstantBooth achieves a remarkable speed improvement of x100 while preserving similar visual quality to existing approaches.

Try the Paper and Project. Don’t forget to hitch our 21k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you’ve any questions regarding the above article or if we missed anything, be at liberty to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club