Free viewpoint video synthesis within the constrained range of an omnidirectional video using DIBR and layered inpainting
摘要
The Depth-Image-based Rendering (DIBR) algorithm is a foundational technology for enabling six-degree-of-freedom immersion in virtual reality (VR). By warping omnidirectional videos (OV) using per-pixel depth data, DIBR achieves real-time novel view synthesis, generating perspectives corresponding to the viewer’s head movements. However, holes frequently appear in regions where sharp depth discontinuities exist, detrimentally affecting the immersive quality of OV. To mitigate these visual artifacts, we propose a layered inpainting method that produces plausible content within holes, ensuring consistency with the scene context. The proposed method incorporates three key technical contributions to tackle this challenge. First, a multi-stage depth estimation pipeline uses existing models to generate high-fidelity and temporally consistent depth videos from OV. Second, a distortion-aware video inpainting model synthesizes disoccluded regions across the entire OV with context/synthesis masks. The model ensures the plausible completion of missing spatial-temporal information, while maintaining geometric and photometric consistency within a constrained motion range. Third, three-layer spherical meshes are constructed from the RGBD video and synthesized occluded elements, which provides real-time novel view synthesis using DIBR (>90 FPS). Comprehensive ablation studies quantify the contribution of each module, with experimental results confirming a significant enhancement in immersion metrics compared to existing methods.