Evaluation of Systematic Errors in Visual Question Answering

Counting questions are considered to be a subfield of the Visual Question Answering (VQA) research area. To evaluate VQA systems properly, a VQA dataset is needed in which all possible answers for all possible counting questions occur equally often. For this purpose, a generator program is developed to create a balanced dataset automatically to help in analyzing the VQA general network architecture and the VQAv2 dataset. The results show that the achieved accuracy of VQAv2 is mostly due to the structure of the questions and answers. On the other hand, when using the generated dataset, the VQA network is not able to achieve an accuracy of more than 12.12%, which is far below the 35.18% in the evaluation of the VQAv2 dataset. We found that two types of information can be exploited by a VQA network in the image to achieve better results: a characteristic object colour and a fixed association of image positions with certain numbers. Our work is a starting point for further work on the analysis of systemic errors in VQA, especially in the area of counting.