Scholarlink

<p>Computational pathology has advanced through foundation models, yet faces challenges in multimodal integration and capturing whole-slide context. Current approaches typically utilize either vision-only or image-caption data, overlooking distinct insights from pathology reports and gene expression profiles. Additionally, most models focus on patch-level analysis, failing to capture comprehensive whole-slide patterns. Here we present mSTAR (<b>M</b>ultimodal <b>S</b>elf-<b>TA</b>ught P<b>R</b>etraining), the pathology foundation model that incorporates three modalities: pathology slides, expert-created reports, and gene expression data, within a unified framework. Our dataset includes 26,169 slide-level modality pairs across 32 cancer types, comprising over 116 million patch images. This approach injects multimodal whole-slide context into patch representations, expanding modeling from single to multiple modalities and from patch-level to slide-level analysis. Across oncological benchmark spanning 97 tasks, mSTAR outperforms previous state-of-the-art models, particularly in molecular prediction and multimodal tasks, revealing that multimodal integration yields greater improvements than simply expanding vision-only datasets.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multimodal knowledge-enhanced whole-slide pathology foundation model

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年12月12日

查看全文
  • Yingxue Xu,
  • Yihui Wang,
  • Fengtao Zhou,
  • Jiabo Ma,
  • Cheng Jin,
  • Shu Yang,
  • Jinbang Li,
  • Zhengyu Zhang,
  • Chenglong Zhao,
  • Huajun Zhou,
  • Zhenhui Li,
  • Huangjing Lin,
  • Xin Wang,
  • Jiguang Wang,
  • Anjia Han,
  • Ronald Cheong Kin Chan,
  • Li Liang,
  • Xiuming Zhang,
  • Hao Chen

摘要

Computational pathology has advanced through foundation models, yet faces challenges in multimodal integration and capturing whole-slide context. Current approaches typically utilize either vision-only or image-caption data, overlooking distinct insights from pathology reports and gene expression profiles. Additionally, most models focus on patch-level analysis, failing to capture comprehensive whole-slide patterns. Here we present mSTAR (Multimodal Self-TAught PRetraining), the pathology foundation model that incorporates three modalities: pathology slides, expert-created reports, and gene expression data, within a unified framework. Our dataset includes 26,169 slide-level modality pairs across 32 cancer types, comprising over 116 million patch images. This approach injects multimodal whole-slide context into patch representations, expanding modeling from single to multiple modalities and from patch-level to slide-level analysis. Across oncological benchmark spanning 97 tasks, mSTAR outperforms previous state-of-the-art models, particularly in molecular prediction and multimodal tasks, revealing that multimodal integration yields greater improvements than simply expanding vision-only datasets.