Reconstructing high-fidelity, controllable 3D human avatars from monocular RGB videos remains a significant challenge for real-time immersive applications such as telepresence and virtual reality. We introduce an enhanced Gaussian-Embedded-Mesh (GEM) framework that advances state-of-the-art physics-aware avatar reconstruction. Our approach anchors anisotropic Gaussian primitives to deformable SMPL mesh triangles, utilizing linear blend skinning (LBS) for accurate pose-driven deformation. A novel physics-based lighting module decomposes surface reflectance into diffuse, specular, and ambient components, enabling precise view-dependent shading. To address non-rigid deformations, we integrate a lightweight pose-refinement network that synergizes with mesh regularization. Joint optimization under a unified training regime–guided by photometric consistency, mesh smoothness, and physics-driven losses–yields substantial improvements in texture fidelity and pose accuracy. Experimental results on the ZJU-MoCap and PeopleSnapshot datasets show that our method achieves 30.68 dB PSNR (novel view) and 30.76 dB PSNR (novel pose) at 55 FPS on an RTX 3090. In terms of various metrics such as PSNR, SSIM, and LPIPS, it outperforms existing methods. This work demonstrates its potential for real-time, physically grounded avatar reconstruction.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Physics-Aware Lighting Gaussian-Embedded-Mesh Avatars from Monocular Video

  • Zhihong Peng,
  • Xinrong Hu,
  • Saishang Zhong,
  • Jinxing Liang,
  • Li Li,
  • Jia Chen,
  • Ziwei Luo

摘要

Reconstructing high-fidelity, controllable 3D human avatars from monocular RGB videos remains a significant challenge for real-time immersive applications such as telepresence and virtual reality. We introduce an enhanced Gaussian-Embedded-Mesh (GEM) framework that advances state-of-the-art physics-aware avatar reconstruction. Our approach anchors anisotropic Gaussian primitives to deformable SMPL mesh triangles, utilizing linear blend skinning (LBS) for accurate pose-driven deformation. A novel physics-based lighting module decomposes surface reflectance into diffuse, specular, and ambient components, enabling precise view-dependent shading. To address non-rigid deformations, we integrate a lightweight pose-refinement network that synergizes with mesh regularization. Joint optimization under a unified training regime–guided by photometric consistency, mesh smoothness, and physics-driven losses–yields substantial improvements in texture fidelity and pose accuracy. Experimental results on the ZJU-MoCap and PeopleSnapshot datasets show that our method achieves 30.68 dB PSNR (novel view) and 30.76 dB PSNR (novel pose) at 55 FPS on an RTX 3090. In terms of various metrics such as PSNR, SSIM, and LPIPS, it outperforms existing methods. This work demonstrates its potential for real-time, physically grounded avatar reconstruction.