<p>Utilizing deep learning to process multi-modal information, including clinical images, dermatoscopy images, and patient metadata, for multi-modal skin lesion diagnosis (MSLD) aligns with modern medical dermatological diagnostic methods. A crucial task in achieving MSLD is to fully leverage multi-modal information, which previous works often failed to accomplish. To address this challenge, in this paper, we propose a novel network, MSfuseNet, for multi-modal skin disease classification. Our method primarily consists of four modules: (1) First, we employ a Coordinate-Spatial Attention Fusion Module to align the two types of images in both coordinate and spatial dimensions, combining this module at different stages with intermediate fusion strategies to reduce information loss in the model. (2) Then, we utilize a MIX Module to facilitate the transformation of shallow local features into deep global features within the model, thereby enhancing the model’s modeling capabilities and robustness. (3) We adopt a Double-Modality Cross-Attention Fusion Module, employing cross-attention mechanisms for global modeling of image features. (4) Finally, we employ a Triple-Modality Fusion Module to aggregate textual features and image features, achieving a full integration of multi-modal information. We have validated the effectiveness of our approach on a public dataset named Derm7pt and a dataset we collected named XJU-MMSD. Compared to state-of-the-art methods, our method achieved the highest average accuracy of 77.73% on the Derm7pt dataset.</p>