MCAF-SkinNet : Multi-modal Cross-Attention Fusion for Skin Disease Classification Using Vision Models and Dermatology-Specific Embeddings
摘要
Accurate classification of skin diseases from clinical images remains a major challenge in medical AI due to variations in skin tones, lesion appearance, lighting conditions, and the scarcity of high-quality labeled datasets for rare conditions. We present MCAF-SkinNet, a novel multimodal framework for skin disease classification that fuses general vision-language representations (SigLIP), biomedical vision-language embeddings (BiomedCLIP), and dermatology-specific expert embeddings (Google Derm Foundation). A bidirectional cross-attention mechanism aligns modality-specific representations, and a learnable gating unit emphasizes clinically salient features. Evaluated on the Fitzpatrick17k dataset spanning 114 skin disease classes, MCAF-SkinNet outperforms strong CNN and Transformer baselines and demonstrates consistent gains across both rare and common conditions. Extensive ablation confirms the complementary nature of each modality and the efficacy of cross-attention fusion.