Detecting surface defects is essential for ensuring quality in industrial production processes, demanding ever-increasing requirements for both precision and real-time performance. However, the pronounced differences in size and morphology among defects, combined with the inherent difficulty in distinguishing small defects from visually similar backgrounds, pose significant challenges to accurate defect identification. To tackle these issues, we introduce DRMNet, a light weight segmentation model that leverages multi-scale features. First, we propose the Multi-Scale Feature Enhancement Module (MFEM), which efficiently captures multi-scale features via dilated convolutions, thereby enhancing the network’s feature extraction capabilities. Furthermore, to address the challenge of distinguishing defects and boundaries, we design the Bilateral Fusion Module (BFM), which directs the network to refine and emphasize the detailed characteristics of defects. Finally, we propose the Deep Parallel Pyramid Pooling Module (DPPPM) is further introduced to fuse multi-scale contextual information, capturing global semantics and local fine details. Experimental results on artificial NRSD, natural NRSD and MT datasets show that DRMNet outperforms existing lightweight segmentation methods.