Physics-Aware Lighting Gaussian-Embedded-Mesh Avatars from Monocular Video
摘要
Reconstructing high-fidelity, controllable 3D human avatars from monocular RGB videos remains a significant challenge for real-time immersive applications such as telepresence and virtual reality. We introduce an enhanced Gaussian-Embedded-Mesh (GEM) framework that advances state-of-the-art physics-aware avatar reconstruction. Our approach anchors anisotropic Gaussian primitives to deformable SMPL mesh triangles, utilizing linear blend skinning (LBS) for accurate pose-driven deformation. A novel physics-based lighting module decomposes surface reflectance into diffuse, specular, and ambient components, enabling precise view-dependent shading. To address non-rigid deformations, we integrate a lightweight pose-refinement network that synergizes with mesh regularization. Joint optimization under a unified training regime–guided by photometric consistency, mesh smoothness, and physics-driven losses–yields substantial improvements in texture fidelity and pose accuracy. Experimental results on the ZJU-MoCap and PeopleSnapshot datasets show that our method achieves 30.68 dB PSNR (novel view) and 30.76 dB PSNR (novel pose) at 55 FPS on an RTX 3090. In terms of various metrics such as PSNR, SSIM, and LPIPS, it outperforms existing methods. This work demonstrates its potential for real-time, physically grounded avatar reconstruction.