VMINer: Versatile Multi-view Inverse Rendering with Near- and Far-field Light Sources

CVPR 2024 Highlight

1Peking University 2The Hong Kong University of Science and Technology 3Light Illusions *Corresponding author

VMINer leverages all present lighting conditions in input images to reconstruct scene geometry and material disentangled with lighting.

Abstract

This paper introduces a versatile multi-view inverse rendering framework with near- and far-field light sources. Tackling the fundamental challenge of inherent ambiguity in inverse rendering, our framework adopts a lightweight yet inclusive lighting model for different near- and far-field lights, thus is able to make use of input images under varied lighting conditions available during capture. It leverages observations under each lighting to disentangle the intrinsic geometry and material from the external lighting, using both neural radiance field rendering and physically-based surface rendering on the 3D implicit fields. After training, the reconstructed scene is extracted to a textured triangle mesh for seamless integration into industrial rendering software for various applications. Quantitatively and qualitatively tested on synthetic and real-world scenes, our method shows superiority to state-of-the-art multi-view inverse rendering methods in both speed and quality.

Video

Pipeline

BibTeX

@inproceedings{VMINer24,
            author       = {Fan Fei and
                            Jiajun Tang and
                            Ping Tan and
                            Boxin Shi},
            title        = {{VMINer}: Versatile Multi-view Inverse Rendering with Near- and Far-field Light Sources},
            booktitle    = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition,
                            {CVPR} 2024, Seattle, WA, USA, June 17-22, 2024},
            pages        = {1--11},
            publisher    = {{IEEE}},
            year         = {2023},
        }