A Novel Occlusion-Aware Vote Cost for Light Field Depth Estimation
IEEE TPAMI 2022
Abstract
Capturing the directions of light by light field cameras powers next-generation immersive multimedia applications. A critical problem in taking advantage of the rich visual information in light field images is depth estimation. Conventional light field depth estimation methods build a cost volume that measures the photo-consistency of pixels refocused to a range of depths, and the highest consistency indicates the correct depth. This strategy works well in most regions but usually generates blurry edges in the estimated depth map due to occlusions. Recent work shows that integrating occlusion models to light field depth estimation can largely reduce blurry edges. However, existing occlusion handling methods rely on complex edge-aided processing and post-refinement, and this reliance limits the resultant depth accuracy and impacts on the computational performance. In this paper, we propose a novel occlusion-aware vote cost (OAVC) which is able to accurately preserve edges in the depth map. Instead of using photo-consistency as an indicator of the correct depth, we construct a novel cost from a new perspective that counts the number of refocused pixels whose deviations from the central-view pixel are less than a small threshold, and utilizes that number to select the correct depth. The pixels from occluders are thus excluded in determining the correct depth. Without the use of any explicit occlusion handling methods, the proposed method can inherently preserve edges and produces high-quality depth estimates. Experimental results show that the proposed OAVC outperforms state-of-the-art light field depth estimation methods in terms of depth estimation accuracy and computational complexity.
Consistency Analysis
By inspecting high-consistency pixels in the scaled refocused lines in Fig. 2, it is found that the number of high-consistency pixels in a correctly refocused line is greater than that of its incorrectly refocused counterpart, even in the presence of occlusions. For instance, the length of the high-consistency (blue) segment of scaled line EC is longer than the high-consistency segment of scaled line KJ. This means that the number of pixels highly consistent with the central-view pixel in a refocused angular patch is effective in selecting the correct disparity when there is occlusion.
The high consistency among correctly refocused patches without occlusion can be demonstrated by comparing the consistency of correctly and incorrectly refocused angular patches. We plot the pixel deviation histogram to demonstrate a threshold does exist that can well distinguish the consistency in correctly refocused angular patches from that attributable to the spatial consistency in incorrectly refocused angular patches. As can be observed from Fig. 3, an evident threshold for the pixel deviation exists, which separates the two distinct scenarios of correct and incorrect refocusing. Based on these observations, we propose the vote cost that separates refocused pixels by use of a threshold and utilizes the number of the separated pixels as an indicator of correct disparity estimation.
Results
Without any explicit occlusion handling, the proposed OAVC produces clean and sharp initial depth estimates. A refinement method (fast weighted median filter) can further smooth the estimated depth and reduce noise.
Citation
@article{han2022novel,
title={A Novel Occlusion-Aware Vote Cost for Light Field Depth Estimation},
author={Han, Kang and Xiang, Wei and Wang, Eric and Huang, Tao},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2022},
volume={44},
number={11},
pages={8022-8035},
doi={10.1109/TPAMI.2021.3105523}
}