Enhancing Image Comprehension for Computer Science Visual Question Answering

Computer science visual question answering is a fundamental task in the intelligent education. However, current models have poor performance in this task. There are mainly two issues in these models. Firstly, they cannot accurately capture the fine-grained objects and relations in images. Secondly, these models lack the computer domain knowledge. To address the issues, we propose an Image Comprehension Enhancing Model. Specifically, it uses object detection technique to capture fine-grained features of images and utilizes Optical Character Recognition (OCR) to transform fine-grained features into the text information. The model adopts the text information to prompt the large language model, which generates the image caption with the computer domain knowledge. The fine-grained features and image caption can enhance the model’s image comprehension and compensate the lack of knowledge. Additionally, the model utilizes the cross-modal attention mechanism to integrate the image features and fine-grained features of the image with the text features. The experimental results on the CSDQA dataset demonstrate that our proposed model outperforms the baselines, and the accuracy improves at least 4.80%.