SAM-NeRF: NeRF-Based 3D Instance Segmentation with Segment Anything Model

Existing NeRF-based instance segmentation methods lift 2D annotated or predicted semantic and instance masks into 3D through radiance field, but still insufficient in segmenting on the unlabeled semantic classes. The Segment Anything Model (SAM) introduces a foundation model for image segmentation, and achieves impressive zero-shot performance. In this paper, we take advantage of SAM and propose SAM-NeRF for 3D instance segmentation agnostic to semantic labels. Unlike existing SAM-based 3D instance segmentation methods which rely on user provided prompts, SAM-NeRF only uses 2D posed images and SAM generated segmentation results to render novel view images with 3D consistent instance masks. To account for the inconsistencies on the instance identifiers across views, we represent the 3D instances with a neural instance field, and propose a mask matching method to lift SAM segmentation results to 3D scene consistently without any semantic supervision. As SAM segments the scene under various granularity, causing divergence cross views, we further propose a merging algorithm to improve the 3D segmentation performance with unified granularity between different views. Without any semantic guidance, our method outperforms the start-of-the-art 2D and 3D supervised panoptic segmentation methods on unlabeled classes in all evaluated datasets.