Enhancing Skin Lesion Classification: A Self-Attention Fusion Approach with Vision Transformer

Automated skin lesion classification is pivotal in modern dermatology, and recent strides in deep learning have shown immense potential in this field. This paper introduces a novel attention mechanism amalgamating various self-attention variants with the Vision Transformer (ViT) architecture to enhance skin lesion classification performance. By integrating Scaled Dot-Product Attention, Multiplicative Attention, and Additive Attention, a unified framework is devised for capturing diverse contextual cues within dermatological images. Extensive experiments are conducted on skin lesion datasets (ISIC 2017) to assess different loss functions, attention mechanisms, and fusion strategies. Results demonstrate that the proposed method significantly enhances classification performance across all metrics, exhibiting a remarkable improvement of over 12% in F1 score compared to the baseline. This approach not only showcases the efficacy of attention mechanisms in dermatological image analysis but also underscores the potential of ViT architecture in advancing automated skin lesion classification, thereby offering promising prospects for improving diagnostic accuracy and patient care in dermatology.