Home Community Meet FreeU: A Novel AI Technique To Enhance Generative Quality Without Additional Training Or Tremendous-tuning

Meet FreeU: A Novel AI Technique To Enhance Generative Quality Without Additional Training Or Tremendous-tuning

0
Meet FreeU: A Novel AI Technique To Enhance Generative Quality Without Additional Training Or Tremendous-tuning

Probabilistic diffusion models, a cutting-edge category of generative models, have turn out to be a critical point within the research landscape, particularly for tasks related to computer vision. Distinct from other classes of generative models, similar to Variational Autoencoder (VAE), Generative Adversarial Networks (GANs), and vector-quantized approaches, diffusion models introduce a novel generative paradigm. These models employ a hard and fast Markov chain to map the latent space, facilitating intricate mappings that capture latent structural complexities inside a dataset. Recently, their impressive generative capabilities, starting from the high level of detail to the range of the generated examples, have pushed groundbreaking advancements in various computer vision applications similar to image synthesis, image editing, image-to-image translation, and text-to-video generation.

The diffusion models consist of two primary components: the diffusion process and the denoising process. Through the diffusion process, Gaussian noise is progressively incorporated into the input data, progressively transforming it into nearly pure Gaussian noise. In contrast, the denoising process goals to get well the unique input data from its noisy state using a sequence of learned inverse diffusion operations. Typically, a U-Net is employed to predict the noise removal iteratively at each denoising step. Existing research predominantly focuses on the usage of pre-trained diffusion U-Nets for downstream applications, with limited exploration of the inner characteristics of the diffusion U-Net.

A joint study from the S-Lab and the Nanyang Technological University departs from the standard application of diffusion models by investigating the effectiveness of the diffusion U-Net within the denoising process. To achieve a deeper understanding of the denoising process, the researchers introduce a paradigm shift towards the Fourier domain to watch the generation strategy of diffusion models—a comparatively unexplored research area. 

The figure above illustrates the progressive denoising process in the highest row, showcasing the generated images at successive iterations. In contrast, the next two rows present the associated low-frequency and high-frequency spatial domain information after the inverse Fourier Transform, corresponding to every respective step. This figure reveals a gradual modulation of low-frequency components, indicating a subdued rate of change, whereas high-frequency components exhibit more pronounced dynamics throughout the denoising process. These findings will be intuitively explained: low-frequency components inherently represent a picture’s global structure and characteristics, encompassing global layouts and smooth colours. Drastic alterations to those components are generally unsuitable in denoising processes as they’ll fundamentally reshape the image’s essence. Then again, high-frequency components capture rapid changes in the photographs, similar to edges and textures, and are highly sensitive to noise. Denoising processes must remove noise while preserving these intricate details.

Considering these observations regarding low-frequency and high-frequency components during denoising, the investigation extends to find out the precise contributions of the U-Net architecture inside the diffusion framework. At each stage of the U-Net decoder, skip features from the skip connections and backbone features are combined. The study reveals that the first backbone of the U-Net plays a big role in denoising, while the skip connections introduce high-frequency features into the decoder module, aiding within the recovery of fine-grained semantic information. Nevertheless, this propagation of high-frequency features can inadvertently weaken the inherent denoising capabilities of the backbone in the course of the inference phase, potentially resulting in the generation of abnormal image details, as depicted in the primary row of Figure 1.

In light of this discovery, the researchers propose a brand new approach known as “FreeU,” which may enhance the standard of generated samples without requiring additional computational overhead from training or fine-tuning. The overview of the framework is reported below.

Through the inference phase, two specialized modulation aspects are introduced to balance the contributions of features from the first backbone and skip connections of the U-Net architecture. The primary factor, generally known as “backbone feature aspects,” is designed to amplify the feature maps of the first backbone, thereby strengthening the denoising process. Nevertheless, it’s observed that the inclusion of backbone feature scaling aspects, while yielding significant improvements, can occasionally end in undesired over-smoothing of textures. To deal with this concern, the second factor, “skip feature scaling aspects,” is introduced to mitigate the issue of texture over-smoothing.

The FreeU framework demonstrates seamless adaptability when integrated with existing diffusion models, including applications like text-to-image generation and text-to-video generation. A comprehensive experimental evaluation of this approach is conducted using foundational models similar to Stable Diffusion, DreamBooth, ReVersion, ModelScope, and Rerender for benchmark comparisons. When FreeU is applied in the course of the inference phase, these models show a noticeable enhancement in the standard of the generated outputs. The visual representation within the illustration below provides evidence of FreeU’s effectiveness in significantly improving each intricate details and the general visual fidelity of the generated images.

This was the summary of FreeU, a novel AI technique that enhances generative models’ output quality without additional training or fine-tuning. Should you have an interest and wish to learn more about it, please be happy to seek advice from the links cited below. 


Try the Paper and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

Should you like our work, you’ll love our newsletter..

We’re also on WhatsApp. Join our AI Channel on Whatsapp..


Daniele Lorenzi received his M.Sc. in ICT for Web and Multimedia Engineering in 2021 from the University of Padua, Italy. He’s a Ph.D. candidate on the Institute of Information Technology (ITEC) on the Alpen-Adria-Universität (AAU) Klagenfurt. He’s currently working within the Christian Doppler Laboratory ATHENA and his research interests include adaptive video streaming, immersive media, machine learning, and QoS/QoE evaluation.


🔥 Meet Retouch4me: A Family of Artificial Intelligence-Powered Plug-Ins for Photography Retouching

LEAVE A REPLY

Please enter your comment!
Please enter your name here