This paper presents a systematic evaluation framework to assess the performance of the Segment Anything Model (SAM) in traffic scene image segmentation under diverse meteorological conditions. Addressing the critical challenge of environmental perception in autonomous driving systems, this study makes three primary contributions. First, we construct a traffic scene dataset comprising 1,350 high-resolution images (1920 × 1080) with pixel-level annotations, covering nine distinct weather-light combinations, including various conditions of rain, fog, and illumination. Second, this paper designs a novel evaluation system that incorporates adaptive preprocessing and optimized prompting strategies to thoroughly examine SAM’s zero-shot segmentation capability. The experimental results demonstrate that SAM performs remarkably well across a variety of complex scenarios, especially in daytime rainy conditions, where the mean Intersection over Union (mIoU) for vehicle segmentation reaches 82.8%. Moreover, SAM maintains robust performance in low-visibility scenarios. This study showcases SAM’s superior generalization ability through the following three aspects: outperforming baseline models across all weather conditions; effectively handling small objects in complex scenes; and possessing real-time processing capabilities.