Seg-invRender: fusing semantic segmentation based on NeRF for inverse rendering considering shadows

Inverse rendering remains a challenging problem in computer vision, but the recent advance in implicit neural rendering methods has introduced new ideas. Most current related works decompose the geometry, illumination and materials of a scene under an unknown lighting environment, but due to lack of explicit constraints on the material, the prediction of materials is easily interfered by shadows which may be misinterpreted as intrinsic materials. To address this issue, we propose an inverse rendering method that incorporates semantic segmentation. Considering obvious shadows in a scene, we aim to effectively mitigate inverse rendering’s ill-posedness and decompose the scene’s geometry, illumination, and materials more accurately. Our method first reconstructs sign distance function of the scene, based on which we can obtain a set of surface points for semantic labels on sparse-view images of the scene. We then project these points onto image plane to generate semantic labels on each view images of the scene, enabling sparsely supervised semantic segmentation of the multi-view images of the 3D scene without additional network training. Finally, considering a certain and limited number of material classes of the scene, we further use the results of semantic segmentation to explicitly constrain the materials prediction. Extensive experiments demonstrate that our method can effectively attenuate the influence of shadows on the material prediction when the scene has obvious shadows, leading to a more accurate prediction of illumination and materials. Our code and data are available at https://github.com/wjn-uo/Seg-InvRender.