Healthcare and elderly-care domains increasingly demand the integration of heterogeneous multimodal information–ranging from medical images and patient records to health-related social data–to enable accurate and timely decision-making. This necessitates advanced multimodal information extraction (MIE) techniques, yet current multimodal large language models (MLLMs) still face challenges of modal imbalance, spurious correlations, and limited robustness due to their reliance on associative rather than causal learning. To address these challenges, we propose causal multimodal prompt tuning (CausalMPT), explicitly introducing causal reasoning into MLLMS to enhance multimodal understanding. CausalMPT adopts a modular design and features three key innovations: a counterfactual intervention module for eliminating false correlations and revealing causal relationships between modalities; The parameter efficient tuning module based on lora can effectively adapt to large-scale models with limited resources. And a prompt answer module, which uses prompts for specific tasks to guide MIE’s fine-grained causal reasoning. A large number of experiments conducted on four benchmark datasets have shown that CausalMPT has achieved consistent improvements. Compared with the basic model, F1 has increased by \(1.47\%\) , \(0.11\%\) , \(1.20\%\) , and \(5.02\%\) respectively.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CausalMPT: Causal Multimodal Prompt Tuning for Healthcare MIE

  • Ziyue Huang,
  • Xiaoming Liu,
  • Guan Yang,
  • Jie Liu,
  • Yang Long,
  • Junyi Gao,
  • Kai Yang

摘要

Healthcare and elderly-care domains increasingly demand the integration of heterogeneous multimodal information–ranging from medical images and patient records to health-related social data–to enable accurate and timely decision-making. This necessitates advanced multimodal information extraction (MIE) techniques, yet current multimodal large language models (MLLMs) still face challenges of modal imbalance, spurious correlations, and limited robustness due to their reliance on associative rather than causal learning. To address these challenges, we propose causal multimodal prompt tuning (CausalMPT), explicitly introducing causal reasoning into MLLMS to enhance multimodal understanding. CausalMPT adopts a modular design and features three key innovations: a counterfactual intervention module for eliminating false correlations and revealing causal relationships between modalities; The parameter efficient tuning module based on lora can effectively adapt to large-scale models with limited resources. And a prompt answer module, which uses prompts for specific tasks to guide MIE’s fine-grained causal reasoning. A large number of experiments conducted on four benchmark datasets have shown that CausalMPT has achieved consistent improvements. Compared with the basic model, F1 has increased by \(1.47\%\) , \(0.11\%\) , \(1.20\%\) , and \(5.02\%\) respectively.