Scholarlink

Background <p>Multimodal foundation models demonstrate remarkable understanding ability to perform accurate molecular representation, including SMILES sequence-text and molecular graph-text representations. Molecular images represent another molecular modality that provides a more intuitive way to capture the spatial structural relationships of molecules. Existing models suffer from integrating image-text modalities for molecular property prediction, largely due to (1) the scarcity of high-quality image-text paired datasets, as existing molecular databases predominantly contain structural representations (SMILES, graphs) with limited descriptive text tailored for visual molecular understanding, and (2) the fundamental semantic gap between molecular images and textual descriptions, where complex visual features must be precisely aligned with domain chemical terminology and functional descriptions.</p> Results <p>In this study, we present a molecular image-text foundation model, named ITMol, pretrained on 500k molecular image-text pairs. Specifically, ITMol addresses the scarcity of high-quality image-text paired datasets by constructing a comprehensive dataset through collection from chemical databases and automated generation using MolT5 to create high-quality image-text pairs, and tackles the fundamental semantic gap between molecular images and textual descriptions by implementing a sophisticated cross-attention mechanism designed specifically for molecular modalities, coupled with three complementary self-supervised learning strategies. We demonstrate the high performance of ITMol in molecular property prediction across 8 benchmark datasets, achieving an ROC-AUC score of 0.885 on the BBBP dataset. ITMol shows high R@5 scores on a dataset of 100k molecules in the cross-modal retrieval task. We further illustrate interpretability of ITMol using key chemical structures and functional groups.</p> Conclusions <p>ITMol provides an effective framework for integrating 2D molecular images and molecular textual descriptions. The results show that image-text multimodal pretraining can improve molecular representation learning for property prediction and retrieval, offering a useful direction for multimodal molecular modeling.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ITMol: a molecular image-text foundation model bridging the semantic gap for property prediction and retrieval

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月18日

查看全文
  • Xuan Lin,
  • Yang Sun,
  • Luming Chen,
  • Xiang Zhang,
  • Lichang Dai,
  • Haowen Chen,
  • Hongxin Xiang,
  • Zuguo Yu

摘要

Background

Multimodal foundation models demonstrate remarkable understanding ability to perform accurate molecular representation, including SMILES sequence-text and molecular graph-text representations. Molecular images represent another molecular modality that provides a more intuitive way to capture the spatial structural relationships of molecules. Existing models suffer from integrating image-text modalities for molecular property prediction, largely due to (1) the scarcity of high-quality image-text paired datasets, as existing molecular databases predominantly contain structural representations (SMILES, graphs) with limited descriptive text tailored for visual molecular understanding, and (2) the fundamental semantic gap between molecular images and textual descriptions, where complex visual features must be precisely aligned with domain chemical terminology and functional descriptions.

Results

In this study, we present a molecular image-text foundation model, named ITMol, pretrained on 500k molecular image-text pairs. Specifically, ITMol addresses the scarcity of high-quality image-text paired datasets by constructing a comprehensive dataset through collection from chemical databases and automated generation using MolT5 to create high-quality image-text pairs, and tackles the fundamental semantic gap between molecular images and textual descriptions by implementing a sophisticated cross-attention mechanism designed specifically for molecular modalities, coupled with three complementary self-supervised learning strategies. We demonstrate the high performance of ITMol in molecular property prediction across 8 benchmark datasets, achieving an ROC-AUC score of 0.885 on the BBBP dataset. ITMol shows high R@5 scores on a dataset of 100k molecules in the cross-modal retrieval task. We further illustrate interpretability of ITMol using key chemical structures and functional groups.

Conclusions

ITMol provides an effective framework for integrating 2D molecular images and molecular textual descriptions. The results show that image-text multimodal pretraining can improve molecular representation learning for property prediction and retrieval, offering a useful direction for multimodal molecular modeling.