RBTFusion: Region-Based Tracking for Real-Time Reconstruction from RGB-D Data

Real-time 3D reconstruction with consumer-level RGB-D cameras is an essential task in the field of augmented reality. Using handheld RGB-D cameras for scene and object depth modeling can assist computers in quickly locating space and achieving virtual-real interaction. Currently, state-of-the-art methods often rely on color features for spatial localization and reconstruction. While these methods achieve impressive results in sequences with rich color features, they struggle to track sequences with weak textures. To tackle this issue, we propose RBTFusion, a real-time 3D reconstruction system that performs spatial registration through region-based implicit segmentation. We employ color distributions and point cloud distances as posterior probabilities, implicitly segmenting both 2D images and 3D space to estimate camera poses. Experimental results on the TUM, Scannet, and our proposed RBTObj dataset demonstrate that our method has a broader applicability range, superior tracking robustness, and better reconstruction accuracy compared to existing methods, particularly in object reconstruction sequences where it exhibits significant advantages. Additionally, RBTFusion exhibits excellent efficiency throughout the tracking and point cloud fusion process, maintaining a stable frame rate of over 35 Hz from tracking to fusion.