RFAConv-CBM-ViT: enhanced vision transformer for metal surface defect detection

Detecting surface defects in metal manufacturing is critical, as they can greatly affect product quality and production efficiency. While vision transformers (ViTs) have advanced in surface defect detection, challenges remain, particularly due to high variability and sample imbalance in metal defects. To address these challenges, this paper proposes a vision transformer model that combines receptive-field attention convolution (RFAConv) and context broadcasting median (CBM), referred to as RFAConv-CBM-ViT. Firstly, this paper introduces RFAConv in the patch embedding stage, which adaptively adjusts the receptive field size, enhancing the model’s ability to capture global features and improving its capability to handle defects of varying scales. Its advantage lies in improving the model’s accuracy and convergence speed without introducing additional computational overhead, while also enhancing its robustness in addressing complex defects. Secondly, this paper design a method called CBM to enhance the performance of the ViT backbone. CBM utilizes median tokens within the multi-layer perceptron to reduce the density of the attention map. By aggregating local contextual information, CBM effectively suppresses noise and irrelevant features, leading to improved model accuracy without increasing computational cost. Experimental results demonstrate that the proposed RFAConv-CBM-ViT model achieves competitive performance across multiple public metal surface defect datasets. This method significantly enhances the performance of vision transformers in metal surface defect detection tasks, demonstrating its potential for applications in the metal manufacturing industry. Code is available at https://github.com/Vzoooong/RFAConv-CBM-ViT/tree/main.