Image restoration is a fancy challenge that has garnered significant attention from researchers. Its primary objective is to create visually appealing and natural images while maintaining the perceptual quality of the degraded input. In cases where there isn’t a information available regarding the subject or degradation (blind restoration), having a transparent understanding of the range of natural images is critical. To revive facial images, it is crucial to incorporate an identity before ensuring that the output retains the person’s unique facial expression. Previous research has looked into using reference-based face image restoration to handle this requirement. Nonetheless, integrating personalization into diffusion-based blind restoration systems stays a persistent challenge.

A team of researchers from the University of California, Los Angeles, and Snap Inc. have developed a way for personalized image restoration called Dual-Pivot Tuning. Dual-Pivot Tuning is an approach used to customize a text-to-image prior within the context of blind image restoration. The method involves utilizing a limited set of high-quality images of a person to reinforce the restoration of their other degraded images. The first objectives are to be certain that the restored images exhibit high fidelity to the person’s identity and the degraded input image while maintaining a natural appearance. 

The study discusses diffusion-based blind restoration methods that may not effectively preserve the unique identity of a person when applied to degraded facial images. The researchers highlight previous efforts in reference-based face image restoration, citing various methods comparable to GFRNet, GWAINet, ASFFNet, Wang et al., DMDNet, and MyStyle. These approaches leverage single or multiple reference images to attain personalized restoration, ensuring higher fidelity to the distinct features of the person within the degraded images. The proposed technique differs from previous methods using a diffusion-based personalized generative prior, while other methods use feedforward architectures or GAN-based priors.

The study outlines the tactic for personalizing guided diffusion models for image restoration. Dual-Pivot Tuning technique involves two steps: text-based fine-tuning to embed identity-specific information inside diffusion priors and model-centric pivoting to harmonize the guiding image encoder with the personalized priors. The personalization operator of text-to-image diffusion models is defined where a model is fine-tuned with a pivot to create a customized version. The technique involves in-context textual pivoting, injecting identity information, followed by model-based pivoting, which utilizes general restoration before achieving high-fidelity restored images.

The proposed Dual-Pivot Tuning technique for personalized restoration achieves high identity fidelity and natural appearance in restored images. Qualitative comparisons show that diffusion-based blind restoration approaches may not retain the person’s identity. At the identical time, the proposed technique maintains high identity fidelity without perceivable loss in fidelity to the degraded input. Quantitative evaluations using metrics comparable to PSNR, SSIM, and ArcFace similarity exhibit the effectiveness of the proposed method in restoring images with high fidelity to the person’s identity.

In conclusion, the proposed technique for personalized restoration via Dual-Pivot Tuning achieves high identity fidelity and natural appearance in restored images. Experiments exhibit the prevalence of the proposed method compared to varied state-of-the-art alternatives for blind and few-shot personalized face image restoration. The customized model shows improved fidelity to the person’s identity and outperforms generic priors regarding general image quality. The tactic is agnostic to various kinds of degradation and provides consistent restoration while retaining identity. 

Try the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to affix our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, LinkedIn Group, Twitter, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

If you happen to like our work, you’ll love our newsletter..