In recent times, there was a growing fascination with the duty of acquiring a 3D generative model from 2D images. With the arrival of Neural Radiance Fields (NeRF), the standard of images produced from a 3D model has witnessed a big advancement, rivaling the photorealism achieved by 2D models. While specific approaches focus solely on 3D representations to make sure consistency within the third dimension, this often comes on the expense of reduced photorealism. Newer studies, nevertheless, have shown that a hybrid approach can overcome this limitation, leading to intensified photorealism. Nonetheless, a notable drawback of those models lies within the intertwining of scene elements, including geometry, appearance, and lighting, which hinders user-defined control. 

Various approaches have been proposed to untangle this complexity. Nonetheless, they demand collections of multiview images of the topic scene for effective implementation. Unfortunately, this requirement poses difficulties when coping with images taken under real-world conditions. While some efforts have relaxed this condition to encompass pictures from different scenes, the need for multiple views of the identical object persists. Moreover, these methods lack generative capabilities and necessitate individual training for every distinct object, rendering them unable to create novel objects. When considering generative methodologies, the interlaced nature of geometry and illumination stays difficult.

The proposed framework, often called FaceLit, introduces a way for acquiring a disentangled 3D representation of a face exclusively from images.

An summary of the architecture is presented within the figure below.

At its core, the approach revolves around constructing a rendering pipeline that enforces adherence to established physical lighting models, much like prior work, tailored to accommodate 3D generative modeling principles. Furthermore, the framework capitalizes on available lighting and pose estimation tools.

The physics-based illumination model is integrated into the recently developed Neural Volume Rendering pipeline, EG3D, which uses tri-plane components to generate deep features from 2D images for volume rendering. Spherical Harmonics are utilized for this integration. Subsequent training focuses on realism, benefiting from the framework’s inherent adherence to physics to generate lifelike images. This alignment with physical principles naturally facilitates the acquisition of a disentangled 3D generative model.

Crucially, the pivotal element enabling the methodology is the combination of physics-based rendering principles into neural volume rendering. As previously indicated, the strategy is designed for seamless integration with pre-existing, available illumination estimators by leveraging Spherical Harmonics. Inside this framework, the diffuse and specular elements of the scene are characterised by Spherical Harmonic coefficients attributed to surface normals and reflectance vectors. These coefficients encompass diffuse reflectance, material specular reflectance, and normal vectors, that are generated through a neural network. This seemingly straightforward setup, nevertheless, effectively untangles illumination from the rendering process.

The proposed approach is implemented and tested across three datasets: FFHQ, CelebA-HQ, and MetFaces. Based on the authors, this yields state-of-the-art FID scores, positioning the tactic on the forefront of 3D-aware generative models. A number of the results produced by the discussed method are reported below.

This was the summary of FaceLit, a brand new AI framework for acquiring a disentangled 3D representation of a face exclusively from images. Should you have an interest and need to learn more about it, please be happy to discuss with the links cited below.

Take a look at the Paper and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.