Uanet: uncertainty-aware cost volume aggregation-based multi-view stereo for 3D reconstruction

Multi-view stereo (MVS) plays a vital role in 3D reconstruction, which aims to reconstruct the 3D point cloud model from multi-view images. Recently, learning-based MVS methods have demonstrated excellent performance compared with traditional MVS methods. Almost all current learning-based MVS methods focus on improving the accuracy and completeness of the reconstruction results. However, scalability remains a major limitation due to the memory constraint. In this paper, a cascaded network with an uncertainty-aware cost volume aggregation named UANet is proposed for efficient and effective dense 3D reconstruction. In particular, we present a novel uncertainty-aware cost volume aggregation approach that takes pair-wise uncertainty maps as guidance to adaptively aggregate cost volumes. Instead of applying 3D convolutional neural networks (CNNs), we introduce the feature difference with a shallow 2D CNN to compute uncertainty maps, which guarantees both efficiency and effectiveness. Moreover, we adopt a coarse-to-fine strategy and use a group-wise correlation to construct lightweight cost volumes, thus reducing the memory consumption and enabling high-resolution depth map inference. Finally, an uncertainty loss is designed to construct the uncertainty map, which can further boost the performance. Experimental results show that UANet outperforms the previous state-of-the-art methods on three benchmarks of DTU benchmark dataset, Tanks and Temples dataset, and BlendedMVS dataset. Besides, the runtime and memory requirements validate the effectiveness of UANet.