Analyzing Difficulties of Visual Questions Using Entropy Values of Answer Predictions

Our approach introduces a novel method for assessing the difficulty of visual questions in Visual Question Answering (VQA) that does not depend on human annotations. Rather than using human-provided responses to real-world situations, as previous studies have done, we evaluate the performance of various VQA models to determine the level of question difficulty. We combine the predicted response distributions of three models: a baseline model that considers both images and questions, and two modified models that incorporate image and question inputs. We group visual questions in the VQA v2 validation set into clusters using k-means clustering and use advanced techniques to calculate accuracy and entropy values for each cluster. Each cluster corresponds to a specific level of visual question complexity, eliminating the need for explicit labeling of difficulty. Our approach effectively identifies clusters of challenging visual questions that even state-of-the-art tools struggle to answer. We conducted a thorough analysis of the VQA v2 dataset and found that all methods performed poorly on the most challenging cluster (with around 10% accuracy), and the accuracy of different models’ predictions diverged as the question difficulty increased. We also observed a strong correlation between entropy and accuracy values for each cluster. Our method can evaluate visual question complexity without depending on ground truth data. By clustering visual questions in the test set, we can inspire the development of new research approaches and techniques.