Current approaches to multimodal survival analysis based on pathology and genomics often apply attention mechanisms at a single stage, thereby missing fine-grained intra-modal details and cross-modal synergies. We propose a novel framework—Multimodal Fusion Framework with Full-stage Attention (MFF-FA)—which integrates attention across three stages: feature extraction, modality alignment, and modality fusion. Specifically, we introduce En-UNI for pathology image sampling and representation, and employ attention modules to enhance latent genomic feature interactions. A convolution-based cross-attention module is further developed to model inter-modal correlations between pathology and genomics. Finally, a gating-attention fusion strategy is applied to retain modality-specific characteristics while enhancing their integration. Comparative evaluations demonstrate that MFF-FA outperforms existing state-of-the-art models in predictive accuracy for cancer survival analysis.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multimodal Fusion Framework Employing Full-Stage Attention for Survival Prediction

  • Aolei Liu,
  • Junkang Guo,
  • Yuhang Wang,
  • Jian Liu,
  • Huiyuan Tian,
  • Tao Qin,
  • Hangrui Xu,
  • Haiyan Wei

摘要

Current approaches to multimodal survival analysis based on pathology and genomics often apply attention mechanisms at a single stage, thereby missing fine-grained intra-modal details and cross-modal synergies. We propose a novel framework—Multimodal Fusion Framework with Full-stage Attention (MFF-FA)—which integrates attention across three stages: feature extraction, modality alignment, and modality fusion. Specifically, we introduce En-UNI for pathology image sampling and representation, and employ attention modules to enhance latent genomic feature interactions. A convolution-based cross-attention module is further developed to model inter-modal correlations between pathology and genomics. Finally, a gating-attention fusion strategy is applied to retain modality-specific characteristics while enhancing their integration. Comparative evaluations demonstrate that MFF-FA outperforms existing state-of-the-art models in predictive accuracy for cancer survival analysis.