Recent advances in Novel-View Synthesis (NVS) and 3D Generation (3DGen) from 2D images have marked significant progress in various domains. While the Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines remain prevalent, their limitations have driven the exploration of Deep Learning (DL)-based methods. Among these, Neural Radiance Fields (NeRFs) stand out for their exceptional capabilities in novel view synthesis and 3D reconstruction. However, their reliance on large, diverse 2D images for training, which capture the same scene from different perspectives, poses challenges. To address these challenges, our research proposes a module that introduces innovative data-centric strategies to improve the fidelity of novel view synthesis and reconstruction of NeRFs. In particular, the adopted strategy relies on depth priors, RGB masks, geometrical warping, and deep learning-based image restoration to improve the training and performance of NeRF models, following a human-in-the-loop approach. This module paves the way for a novel data-centric and DL-driven, to improve performances in NeRFs, which is adaptable across different NeRF architectures. Through a comprehensive quantitative-qualitative analysis of such a framework, on a challenging NeRF benchmark dataset, we demonstrate the effectiveness and versatility of our approach.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Data-Centric Module for Neural Rendering

  • Emanuele Balloni,
  • Lorenzo Stacchio,
  • Lucrezia Gorgoglione,
  • Marina Paolanti,
  • Roberto Pierdicca,
  • Adriano Mancini,
  • Emanuele Frontoni,
  • Primo Zingaretti

摘要

Recent advances in Novel-View Synthesis (NVS) and 3D Generation (3DGen) from 2D images have marked significant progress in various domains. While the Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines remain prevalent, their limitations have driven the exploration of Deep Learning (DL)-based methods. Among these, Neural Radiance Fields (NeRFs) stand out for their exceptional capabilities in novel view synthesis and 3D reconstruction. However, their reliance on large, diverse 2D images for training, which capture the same scene from different perspectives, poses challenges. To address these challenges, our research proposes a module that introduces innovative data-centric strategies to improve the fidelity of novel view synthesis and reconstruction of NeRFs. In particular, the adopted strategy relies on depth priors, RGB masks, geometrical warping, and deep learning-based image restoration to improve the training and performance of NeRF models, following a human-in-the-loop approach. This module paves the way for a novel data-centric and DL-driven, to improve performances in NeRFs, which is adaptable across different NeRF architectures. Through a comprehensive quantitative-qualitative analysis of such a framework, on a challenging NeRF benchmark dataset, we demonstrate the effectiveness and versatility of our approach.