Home Community MoMA: An Open-Vocabulary and Training Free Personalized Image Model that Boasts Flexible Zero-Shot Capabilities

MoMA: An Open-Vocabulary and Training Free Personalized Image Model that Boasts Flexible Zero-Shot Capabilities

0
MoMA: An Open-Vocabulary and Training Free Personalized Image Model that Boasts Flexible Zero-Shot Capabilities

Modern image-generating tools have come a good distance because of large-scale text-to-image diffusion models like GLIDE, DALL-E 2, Imagen, Stable Diffusion, and eDiff-I. Due to these models, users can create realistic pictures using a wide range of textual cues. Kandinsky and Stable Unclip take images as inputs to generate variations that retain the visual components of the reference. The emergence of image-conditioned generation works like Kandinsky, and Stable Unclip is a response to the incontrovertible fact that textual descriptions, while effective, incessantly fail to convey detailed visual features.

Image personalization or subject-driven generation is the subsequent logical step on this area. Early attempts on this field include using learnable text tokens to represent goal concepts and converting input photos to text. Nevertheless, the substantial resources needed for instance-specific tuning and model storage severely restrict the practicality of those approaches despite their accuracy. To beat these constraints, tuning-free methods have grow to be more popular. Despite their efficacy in modifying textures, these methods incessantly produce tuning-free detail defects and necessitate further tuning to attain ideal results with goal objects.

A recent study by ByteDance and Rutgers University presents a brand new model called MoMA for picture personalization that doesn’t require tweaking and uses an open vocabulary. It overcomes these issues by effectively integrating logical textual prompts, achieving excellent detail fidelity, and resembling object identities. MoMA for text-to-image diffusion model rapid picture customization.

This approach consists of three parts:

  1. First, the researchers use a generative multimodal decoder to retrieve the reference picture’s features. Then, they alter them in keeping with the goal prompt to get the contextualized image feature. 
  2. Meanwhile, they use the unique UNet’s self-attention layers to extract the article image feature by replacing the background of the unique image with white color and leaving only the article’s pixels. 
  3. Lastly, they used the UNet diffusion model with the object-cross-attention layers and the contextualized picture attributes to generate latest images. The layers were trained specifically for this purpose.

The team used the OpenImage-V7 dataset to construct a dataset of 282K image/caption/image-mask triplets for model training. After generating image captions using BLIP-2 OPT6.7B, any subjects pertaining to humans, color, form, and texture keywords were eliminated.

The experimental results speak volumes concerning the MoMA model’s superiority. By harnessing the ability of Multimodal Large Language Models (MLLMs), the model seamlessly combines the visual characteristics of the goal object with text prompts, enabling changes to each the backdrop context and object texture. The suggested self-attention shortcut significantly enhances detail quality while imposing a minimal computational burden. The model’s expanded applicability is a testament to its potential, as it may well be directly integrated with community models which have been fine-tuned using the identical basic model, opening up latest possibilities in the sector of image generation and machine learning. 


Try the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

In the event you like our work, you’ll love our newsletter..

Don’t Forget to affix our 40k+ ML SubReddit


Dhanshree

” data-medium-file=”https://www.marktechpost.com/wp-content/uploads/2022/11/20221028_101632-Dhanshree-Shenwai-169×300.jpg” data-large-file=”https://www.marktechpost.com/wp-content/uploads/2022/11/20221028_101632-Dhanshree-Shenwai-576×1024.jpg”>

Dhanshree Shenwai is a Computer Science Engineer and has experience in FinTech corporations covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is keen about exploring latest technologies and advancements in today’s evolving world making everyone’s life easy.


🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and lots of others…

LEAVE A REPLY

Please enter your comment!
Please enter your name here