The detection of surface damages is vital for ensuring the safety and longevity of infrastructure, as early identification prevents further degradation and reduces maintenance costs. In autonomous vehicles, accurate surface damage detection is critical for enabling safe navigation, minimizing accidents, and maintaining smooth and efficient transportation systems. This paper presents a lightweight Convolutional Neural Network designed for real-time, accurate semantic segmentation. Unlike recent lightweight networks that favor shallow architectures, the proposed model employs a deeper network structure while maintaining fast inference speed and high segmentation accuracy. The network leverages factorized dilated depth-wise separable convolutions to capture feature representations across multiple scale receptive fields with reduced model parameters. Furthermore, it incorporates multiple branches of skip connections to extract contextual information from intermediate convolution layers. Extensive experiments demonstrate that the proposed model achieves state-of-the-art performance in both speed and accuracy.