<p>Recent advances in deep learning models for object detection and segmentation have received a lot of attention in various industry applications. However, applying deep learning methods on real world problems has additional challenges for acquiring datasets and achieving sufficient performance which meets industrial needs. In steel manufacturing industry, unexpected factors cause critical effects on the quality of steel products, and it is required to inspect defects in an early stage to reduce production costs. This paper proposes TAG-Net, a novel attention-based semantic segmentation network aimed at improving the performance for inspecting surface defects on steel products. TAG-Net estimates three attention maps each for background, defects, and boundaries of defects, and we introduce an auxiliary deep supervision to guide the boundaries of defective regions. Experiments were conducted on the NEU-Seg dataset, and experimental results demonstrate that our proposed method significantly outperforms previous methods with a significant margin.</p>