The LAVA challenge presents complex visual question answering tasks involving intricate diagrams, each accompanied by multiple-choice questions in English or Japanese. Addressing this challenge, we - the team v1olet - explore the capabilities of pre-trained Large Vision-Language Models to interpret and reason over such sophisticated visual data. We utilize models including Qwen2-VL, InternVL2, MiniCPM, and Llama-3.2-Vision-Instruct, employing a structured prompt template designed to standardize response generation and facilitate step-by-step reasoning. To enhance accuracy and robustness, we implement an ensemble method using majority voting to combine outputs from different models and configurations. Our experimental results demonstrate that the ensemble approach significantly improves performance, achieving a higher public score on the LAVA challenge dataset compared to individual models. Specifically, the ensemble of Qwen2-VL, InternVL2, and Llama-3.2 models attained the highest public score of 82, outperforming the best single model. This study highlights the effectiveness of combining multiple Large Vision-Language Models through ensemble methods and underscores the potential of prompt-based inference in enhancing model reasoning capabilities for complex VQA tasks. The provided code is here .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Visual Question Answering with Pre-trained Vision-Language Models: An Ensemble Approach at the LAVA Challenge 2024

  • Trong-Hieu Nguyen-Mau,
  • Nhu-Binh Nguyen Truc,
  • Nhu-Vinh Hoang,
  • Minh-Triet Tran,
  • Hai-Dang Nguyen

摘要

The LAVA challenge presents complex visual question answering tasks involving intricate diagrams, each accompanied by multiple-choice questions in English or Japanese. Addressing this challenge, we - the team v1olet - explore the capabilities of pre-trained Large Vision-Language Models to interpret and reason over such sophisticated visual data. We utilize models including Qwen2-VL, InternVL2, MiniCPM, and Llama-3.2-Vision-Instruct, employing a structured prompt template designed to standardize response generation and facilitate step-by-step reasoning. To enhance accuracy and robustness, we implement an ensemble method using majority voting to combine outputs from different models and configurations. Our experimental results demonstrate that the ensemble approach significantly improves performance, achieving a higher public score on the LAVA challenge dataset compared to individual models. Specifically, the ensemble of Qwen2-VL, InternVL2, and Llama-3.2 models attained the highest public score of 82, outperforming the best single model. This study highlights the effectiveness of combining multiple Large Vision-Language Models through ensemble methods and underscores the potential of prompt-based inference in enhancing model reasoning capabilities for complex VQA tasks. The provided code is here .