There was enduring interest within the manipulation of images because of its big selection of applications in content creation. One of the crucial extensively studied manipulations is object removal and insertion, sometimes called the image inpainting task. While current inpainting models are proficient at generating visually convincing content that blends seamlessly with the encompassing image, their applicability has traditionally been limited to single 2D image inputs. Nonetheless, some researchers try to advance the appliance of such models to the manipulation of complete 3D scenes.

The emergence of Neural Radiance Fields (NeRFs) has made the transformation of real 2D photos into lifelike 3D representations more accessible. As algorithmic enhancements proceed and computational demands decrease, these 3D representations may turn into commonplace. Due to this fact, the research goals to enable similar manipulations of 3D NeRFs as can be found for 2D images, with a selected concentrate on inpainting.

The inpainting of 3D objects presents unique challenges, including the scarcity of 3D data and the need to contemplate each 3D geometry and appearance. Using NeRFs as a scene representation introduces additional complexities. The implicit nature of neural representations makes it impractical to directly modify the underlying data structure based on geometric understanding. Moreover, because NeRFs are trained from images, maintaining consistency across multiple views poses challenges. Independent inpainting of individual constituent images can result in inconsistencies in viewpoints and visually unrealistic outputs.

Various approaches have been attempted to deal with these challenges. For instance, some methods aim to resolve inconsistencies post hoc, equivalent to NeRF-In, which mixes views through pixel-wise loss, or SPIn-NeRF, which employs a perceptual loss. Nonetheless, these approaches may struggle when inpainted views exhibit significant perceptual differences or involve complex appearances.

Alternatively, single-reference inpainting methods have been explored, which avoid view inconsistencies through the use of just one inpainted view. Nonetheless, this approach introduces several challenges, including reduced visual quality in non-reference views, a scarcity of view-dependent effects, and issues with disocclusions.

Considering the mentioned limitations, a brand new approach has been developed to enable the inpainting of 3D objects.

Inputs to the system are N images from different perspectives with their corresponding camera transformation matrices and masks, delineating the unwanted regions. Moreover, an inpainted reference view related to the input images is required, which provides the knowledge that a user expects to assemble from a 3D inpainting of the scene. This reference will be so simple as a text description of the item to interchange the mask.

In the instance reported above, the “rubber duck” or “flower pot” references will be obtained by employing a single-image text-conditioned inpainter. This manner, any user can control and drive the generation of 3D scenes with the specified edits. 

With a module specializing in view-dependent effects (VDEs), the authors attempt to account for view-dependent changes (e.g., specularities and non-Lambertian effects) within the scene. For that reason, they add VDEs to the masked area from non-reference viewpoints by correcting reference colours to match the encompassing context of the opposite views.

Moreover, they introduce monocular depth estimators to guide the geometry of the inpainted region based on the depth of the reference image. Since not all of the masked goal pixels are visible within the reference, an approach is devised to supervise such unoccluded pixels via additional inpaintings.

A visible comparison of novel view renderings of the proposed method with the state-of-the-art SPIn-NeRF-Lama is provided below.

This was the summary of a novel AI framework for reference-guided controllable inpainting of neural radiance fields. Should you have an interest and need to learn more about it, please be happy to check with the links cited below. 

Try the Paper and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 30k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.

Should you like our work, you’ll love our newsletter..