Accurate classification of skin diseases from clinical images remains a major challenge in medical AI due to variations in skin tones, lesion appearance, lighting conditions, and the scarcity of high-quality labeled datasets for rare conditions. We present MCAF-SkinNet, a novel multimodal framework for skin disease classification that fuses general vision-language representations (SigLIP), biomedical vision-language embeddings (BiomedCLIP), and dermatology-specific expert embeddings (Google Derm Foundation). A bidirectional cross-attention mechanism aligns modality-specific representations, and a learnable gating unit emphasizes clinically salient features. Evaluated on the Fitzpatrick17k dataset spanning 114 skin disease classes, MCAF-SkinNet outperforms strong CNN and Transformer baselines and demonstrates consistent gains across both rare and common conditions. Extensive ablation confirms the complementary nature of each modality and the efficacy of cross-attention fusion.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MCAF-SkinNet : Multi-modal Cross-Attention Fusion for Skin Disease Classification Using Vision Models and Dermatology-Specific Embeddings

  • Routhu Srinivasa Rao,
  • Aradhana Mishra,
  • Sanjay Swain

摘要

Accurate classification of skin diseases from clinical images remains a major challenge in medical AI due to variations in skin tones, lesion appearance, lighting conditions, and the scarcity of high-quality labeled datasets for rare conditions. We present MCAF-SkinNet, a novel multimodal framework for skin disease classification that fuses general vision-language representations (SigLIP), biomedical vision-language embeddings (BiomedCLIP), and dermatology-specific expert embeddings (Google Derm Foundation). A bidirectional cross-attention mechanism aligns modality-specific representations, and a learnable gating unit emphasizes clinically salient features. Evaluated on the Fitzpatrick17k dataset spanning 114 skin disease classes, MCAF-SkinNet outperforms strong CNN and Transformer baselines and demonstrates consistent gains across both rare and common conditions. Extensive ablation confirms the complementary nature of each modality and the efficacy of cross-attention fusion.