Scholarlink

In order to solve the problem that the model lacks the logical interpretability of the problem in the visual question answering (VQA) tasks, this paper proposes a new method based on CLIP feature fusion and nearest neighbor context enhancement. The CLIP model is employed to extract visual features from 40k training images in MME-RealWorld and 18k structured samples in M3COT, while text semantic features are derived from question-answer-Chain of Thought (CoT) quadruples. We construct a multi-modal vector space by fusing image-text features with a learnable weight matrix, and optimize the fusion coefficient α via cross-entropy loss on 50k + training samples. In the inference stage, the nearest neighbor retrieval technology is utilized to obtain similar solution cases of test examples, and the current problem is jointly input into the large model for joint inference. Experimental results on MME-RealWorld (40k samples) and M3COT (18k samples) demonstrate accuracy improvements of 4.9% (MME-RealWorld) and 3.7% (M3COT), respectively. The ROUGE-L score increases by 23.7% (64.9% vs. 41.2%), verifying the effectiveness of our context retrieval mechanism across 10 + ablation tests. By converting visual semantics and logical reasoning processes into computable vector representations, our approach provides an interpretable framework for complex VQA tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-Modal Prompt Generation with Chain of Thought for Visual Question Answering

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Guang Yang,
  • Yan Chen,
  • Haoying Sun,
  • Li Fang Wu,
  • Shuyi Li

摘要

In order to solve the problem that the model lacks the logical interpretability of the problem in the visual question answering (VQA) tasks, this paper proposes a new method based on CLIP feature fusion and nearest neighbor context enhancement. The CLIP model is employed to extract visual features from 40k training images in MME-RealWorld and 18k structured samples in M3COT, while text semantic features are derived from question-answer-Chain of Thought (CoT) quadruples. We construct a multi-modal vector space by fusing image-text features with a learnable weight matrix, and optimize the fusion coefficient α via cross-entropy loss on 50k + training samples. In the inference stage, the nearest neighbor retrieval technology is utilized to obtain similar solution cases of test examples, and the current problem is jointly input into the large model for joint inference. Experimental results on MME-RealWorld (40k samples) and M3COT (18k samples) demonstrate accuracy improvements of 4.9% (MME-RealWorld) and 3.7% (M3COT), respectively. The ROUGE-L score increases by 23.7% (64.9% vs. 41.2%), verifying the effectiveness of our context retrieval mechanism across 10 + ablation tests. By converting visual semantics and logical reasoning processes into computable vector representations, our approach provides an interpretable framework for complex VQA tasks.