Scholarlink

<p>Generating captions for diagrams is a challenging task because of their complex structures, diverse visual elements, and domain-specific semantic content. Unlike natural images, diagrams require a detailed understanding of the interplay between visual and textual components. Models such as BLIP-2, MiniGPT-4, and LLaVA are well-suited for general multimodal tasks. However, when applied to the specific demands of diagram captioning, they require longer inference times, making them less practical for this specialized domain. To address these challenges, we propose a framework that integrates a Mixture of Experts (MoE) with Low-Rank Adaptation (LoRA), trained on the AI2D-Caption dataset–comprising expert annotated scientific diagrams from biology, astronomy, and engineering–to achieve resource-efficient and high-quality caption generation. The framework operates in two stages. During the pretraining stage, the model aligns visual and textual modalities into a unified multimodal representation. In the fine-tuning stage, this representation is dynamically routed to domain-specific experts fine-tuned with LoRA. The gating network selectively activates relevant experts. Our approach outperforms the zero-shot performance of existing vision–language models, achieving an average BLEU score of 0.42 and an average SPICE score of 0.38, while maintaining an efficient average inference time of 3 seconds. Our contribution lies in proposing a new benchmark for diagram captioning. Our model using fewer parameters is able to outperform competing models.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A framework for efficient scientific diagram captioning using mixture-of-experts and low-rank adaptation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月11日

查看全文
  • Deepika Kamboj,
  • Gaurav Harit

摘要

Generating captions for diagrams is a challenging task because of their complex structures, diverse visual elements, and domain-specific semantic content. Unlike natural images, diagrams require a detailed understanding of the interplay between visual and textual components. Models such as BLIP-2, MiniGPT-4, and LLaVA are well-suited for general multimodal tasks. However, when applied to the specific demands of diagram captioning, they require longer inference times, making them less practical for this specialized domain. To address these challenges, we propose a framework that integrates a Mixture of Experts (MoE) with Low-Rank Adaptation (LoRA), trained on the AI2D-Caption dataset–comprising expert annotated scientific diagrams from biology, astronomy, and engineering–to achieve resource-efficient and high-quality caption generation. The framework operates in two stages. During the pretraining stage, the model aligns visual and textual modalities into a unified multimodal representation. In the fine-tuning stage, this representation is dynamically routed to domain-specific experts fine-tuned with LoRA. The gating network selectively activates relevant experts. Our approach outperforms the zero-shot performance of existing vision–language models, achieving an average BLEU score of 0.42 and an average SPICE score of 0.38, while maintaining an efficient average inference time of 3 seconds. Our contribution lies in proposing a new benchmark for diagram captioning. Our model using fewer parameters is able to outperform competing models.