Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis

Abstract

Rendering novel views from captured multi-view images has made considerable progress since the emergence of the neural radiance field. This paper aims to further advance the quality of view synthesis by proposing a novel approach dubbed the neural radiance feature field (NRFF). We first propose a multiscale tensor decomposition scheme to organize learnable features so as to represent scenes from coarse to fine scales. We demonstrate many benefits of the proposed multiscale representation, including more accurate scene shape and appearance reconstruction, and faster convergence compared with the single-scale representation. Instead of encoding view directions to model view-dependent effects, we further propose to encode the rendering equation in the feature space by employing the anisotropic spherical Gaussian mixture predicted from the proposed multiscale representation. The proposed NRFF improves state-of-the-art rendering results by over 1 dB in PSNR on both the NeRF and NSVF synthetic datasets. A significant improvement has also been observed on the real-world Tanks & Temples dataset.

Multiscale Tensor Decomposition

The proposed multiscale tensor decomposition representation. At each level, a 3D tensor representation is decomposed to three sets of plane feature maps and line feature vectors. The resolution of decomposed tensors increases with the level, enabling scene representation at different scales. The concatenated feature vectors from all levels are used to predict parameters P (will be used to encode the rendering equation) by a spatial MLP.

Performance comparison over training steps for varying the number of scale levels on scene ship. L indicates the number of levels. All models with different levels have roughly the same number of learnable features. Models with more levels not only converge faster but yield better final PSNRs.

Rendering Equation Encoding

Results

Citation

@inproceedings{han2023nrff,
        title={Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis},
        author={Han, Kang and Xiang, Wei},
        booktitle={The IEEE / CVF Computer Vision and Pattern Recognition Conference},
        pages={4232--4241},
        year={2023}
}