<p>The Depth-Image-based Rendering (DIBR) algorithm is a foundational technology for enabling six-degree-of-freedom immersion in virtual reality (VR). By warping omnidirectional videos (OV) using per-pixel depth data, DIBR achieves real-time novel view synthesis, generating perspectives corresponding to the viewer’s head movements. However, holes frequently appear in regions where sharp depth discontinuities exist, detrimentally affecting the immersive quality of OV. To mitigate these visual artifacts, we propose a layered inpainting method that produces plausible content within holes, ensuring consistency with the scene context. The proposed method incorporates three key technical contributions to tackle this challenge. First, a multi-stage depth estimation pipeline uses existing models to generate high-fidelity and temporally consistent depth videos from OV. Second, a distortion-aware video inpainting model synthesizes disoccluded regions across the entire OV with context/synthesis masks. The model ensures the plausible completion of missing spatial-temporal information, while maintaining geometric and photometric consistency within a constrained motion range. Third, three-layer spherical meshes are constructed from the RGBD video and synthesized occluded elements, which provides real-time novel view synthesis using DIBR (&gt;90 FPS). Comprehensive ablation studies quantify the contribution of each module, with experimental results confirming a significant enhancement in immersion metrics compared to existing methods.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Free viewpoint video synthesis within the constrained range of an omnidirectional video using DIBR and layered inpainting

  • Yunbiao Liu,
  • Chunyi Chen,
  • Haiyang Yu,
  • Jun Peng,
  • Zhongye Shen

摘要

The Depth-Image-based Rendering (DIBR) algorithm is a foundational technology for enabling six-degree-of-freedom immersion in virtual reality (VR). By warping omnidirectional videos (OV) using per-pixel depth data, DIBR achieves real-time novel view synthesis, generating perspectives corresponding to the viewer’s head movements. However, holes frequently appear in regions where sharp depth discontinuities exist, detrimentally affecting the immersive quality of OV. To mitigate these visual artifacts, we propose a layered inpainting method that produces plausible content within holes, ensuring consistency with the scene context. The proposed method incorporates three key technical contributions to tackle this challenge. First, a multi-stage depth estimation pipeline uses existing models to generate high-fidelity and temporally consistent depth videos from OV. Second, a distortion-aware video inpainting model synthesizes disoccluded regions across the entire OV with context/synthesis masks. The model ensures the plausible completion of missing spatial-temporal information, while maintaining geometric and photometric consistency within a constrained motion range. Third, three-layer spherical meshes are constructed from the RGBD video and synthesized occluded elements, which provides real-time novel view synthesis using DIBR (>90 FPS). Comprehensive ablation studies quantify the contribution of each module, with experimental results confirming a significant enhancement in immersion metrics compared to existing methods.