Scholarlink

<p>Vision–language models (VLMs) have recently shown strong potential for multimodal scientific applications, yet their direct application to molecular structure images remains challenging due to the need for precise visual–textual alignment. In this study, we investigate efficient fine-tuning strategies for adapting a large-scale VLM to molecular image captioning tasks. Our approach is built upon the Qwen3.5—9B architecture and focuses on lightweight parameter-efficient adaptation.</p><p>Specifically, we apply Low-Rank Adaptation (LoRA) to the language-side MLP layers and explore selective tuning of the visual merger module that bridges the vision encoder and the language model. This module compresses visual features and projects them into the language embedding space, making it a critical component for cross-modal alignment. Experimental results on molecular image captioning tasks show that the combined L-MLP + Visual Merger tuning strategy achieves the best performance while maintaining low computational overhead.</p><p>In addition, we evaluate multiple decoding strategies, including greedy decoding and stochastic sampling methods, to identify the most suitable inference configuration for molecular image captioning. The results demonstrate that selective fine-tuning of the visual–language interface significantly improves caption accuracy while preserving efficiency. These findings highlight the importance of adapting cross-modal projection modules when deploying general-purpose VLMs in specialized scientific domains such as cheminformatics.</p><p><b>Scientific contribution</b></p><p>We provide a systematic analysis of layer-wise fine-tuning strategies in large-scale vision–language models(VLMs), identifying which components are most critical for adapting general-purpose models to molecular image captioning tasks.</p><p>We identify the cross-modal projection module (visual merger) as a key factor in visual–textual alignment, demonstrating that its selective adaptation plays a central role in improving caption quality.</p><p>We propose a parameter-efficient fine-tuning strategy that combines light weight language-side adaptation with targeted tuning of the visual–language interface, achieving an effective balance between performance and computational cost.</p><p>We provide practical insights into efficient adaptation of VLMs in scientific domains, highlighting key considerations for achieving performance under computational constraints.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Efficient fine-tuning of vision-language adapters in chemical VLMs for molecular image-text tasks

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月28日

查看全文
  • Hyukjun Choi,
  • Donghyeon Lee,
  • Juyoung Kang

摘要

Vision–language models (VLMs) have recently shown strong potential for multimodal scientific applications, yet their direct application to molecular structure images remains challenging due to the need for precise visual–textual alignment. In this study, we investigate efficient fine-tuning strategies for adapting a large-scale VLM to molecular image captioning tasks. Our approach is built upon the Qwen3.5—9B architecture and focuses on lightweight parameter-efficient adaptation.

Specifically, we apply Low-Rank Adaptation (LoRA) to the language-side MLP layers and explore selective tuning of the visual merger module that bridges the vision encoder and the language model. This module compresses visual features and projects them into the language embedding space, making it a critical component for cross-modal alignment. Experimental results on molecular image captioning tasks show that the combined L-MLP + Visual Merger tuning strategy achieves the best performance while maintaining low computational overhead.

In addition, we evaluate multiple decoding strategies, including greedy decoding and stochastic sampling methods, to identify the most suitable inference configuration for molecular image captioning. The results demonstrate that selective fine-tuning of the visual–language interface significantly improves caption accuracy while preserving efficiency. These findings highlight the importance of adapting cross-modal projection modules when deploying general-purpose VLMs in specialized scientific domains such as cheminformatics.

Scientific contribution

We provide a systematic analysis of layer-wise fine-tuning strategies in large-scale vision–language models(VLMs), identifying which components are most critical for adapting general-purpose models to molecular image captioning tasks.

We identify the cross-modal projection module (visual merger) as a key factor in visual–textual alignment, demonstrating that its selective adaptation plays a central role in improving caption quality.

We propose a parameter-efficient fine-tuning strategy that combines light weight language-side adaptation with targeted tuning of the visual–language interface, achieving an effective balance between performance and computational cost.

We provide practical insights into efficient adaptation of VLMs in scientific domains, highlighting key considerations for achieving performance under computational constraints.