We propose Self-Adapting NeRF for high-quality novel view synthesis based on non-ideal video input. We first using Lie algebra to encode camera poses, which were dynamically controlled by parameters from the precomputed Fourier transform encoding in the NeRF network input, achieving joint optimization of poses and the model. Subsequently, by monitoring intermediate training results, we supplement areas with poor performance, implementing a training strategy based on keyframe supplementation and gradient prioritization. This addresses the challenge of achieving high-quality novel view synthesis with NeRF-series in non-ideal input scenarios. Finally, we employ a strategy of geometry and material information separation, along with a reflection lighting model, to address issues in scenes with specular reflections.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Self-Adapting NeRF: Non-ideal Video Based NeRF for High-Quality Novel View Synthesis

  • Tao Huang,
  • Dengming Zhu,
  • Min Shi,
  • Zhaoqi Wang

摘要

We propose Self-Adapting NeRF for high-quality novel view synthesis based on non-ideal video input. We first using Lie algebra to encode camera poses, which were dynamically controlled by parameters from the precomputed Fourier transform encoding in the NeRF network input, achieving joint optimization of poses and the model. Subsequently, by monitoring intermediate training results, we supplement areas with poor performance, implementing a training strategy based on keyframe supplementation and gradient prioritization. This addresses the challenge of achieving high-quality novel view synthesis with NeRF-series in non-ideal input scenarios. Finally, we employ a strategy of geometry and material information separation, along with a reflection lighting model, to address issues in scenes with specular reflections.