While NeRF is a groundbreaking method in the field of scene reconstruction, it faces challenges when dealing with the data characterized by varying occlusions and shadows. To overcome the limitations of NeRFs in occlusion removal and shadow mitigation, we propose a shadow-casting object removal framework based on the Segment Anything Model (SAM) and associate it with NeRF. Specifically, we first introduce a prompt fusion method to effectively mix point and text prompts, guiding the vanilla SAM to better capture the masking edges. Another fine-tuned SAM incorporates with an enhanced edge extraction that leverages consistency in texture and color across the same material to improve the removal of shadows cast by objects within the scene. By combining the refined object mask with shadow-insensitive masks, our model significantly enhance the scene rendering quality, particularly when handling occluded objects. Comprehensive quantitative and qualitative results demonstrate that the proposed framework effectively addresses geometric alignment, color consistency, and texture fidelity, achieving superior performance in object removal and shadow mitigation tasks for NeRFs.