Scholarlink

In the medical field, multimodal data fusion is a key technology for improving diagnostic accuracy. Traditional multimodal models often adopt simplistic approaches to modality fusion, failing to fully exploit and integrate the rich information from different modalities. Additionally, existing Transformer-based fusion methods frequently exhibit poor extensibility when handling more than two modalities. To address this gap, we propose a novel Multimodal Data Fusion Transformer module (MFTrans). MFTrans consists of two main components: the multimodal fusion attention mechanism (MFAttention) and grouped convolutions. MFTrans can simultaneously capture both intra-modality and inter-modality correlations, as well as local information from different modalities, and is highly extensible. We further develop a Multimodal Diagnostic Transformer (MDFormer) based on MFTrans. Additionally, we construct a multimodal chest disease multi-label classification dataset, MIMIC-MCC. Experimental results on the MIMIC-MCC dataset demonstrate that MFTrans achieves state-of-the-art performance across several evaluation metrics compared to existing multimodal fusion methods, showcasing its potential and advantages in practical clinical diagnostic tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MFTrans: An Extensible Transformer-Based Medical Multimodal Data Fusion Method for Clinical Diagnosis

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Xinlong Liu,
  • Chunping Li

摘要

In the medical field, multimodal data fusion is a key technology for improving diagnostic accuracy. Traditional multimodal models often adopt simplistic approaches to modality fusion, failing to fully exploit and integrate the rich information from different modalities. Additionally, existing Transformer-based fusion methods frequently exhibit poor extensibility when handling more than two modalities. To address this gap, we propose a novel Multimodal Data Fusion Transformer module (MFTrans). MFTrans consists of two main components: the multimodal fusion attention mechanism (MFAttention) and grouped convolutions. MFTrans can simultaneously capture both intra-modality and inter-modality correlations, as well as local information from different modalities, and is highly extensible. We further develop a Multimodal Diagnostic Transformer (MDFormer) based on MFTrans. Additionally, we construct a multimodal chest disease multi-label classification dataset, MIMIC-MCC. Experimental results on the MIMIC-MCC dataset demonstrate that MFTrans achieves state-of-the-art performance across several evaluation metrics compared to existing multimodal fusion methods, showcasing its potential and advantages in practical clinical diagnostic tasks.