Visual Question Answering – VizWiz Challenge

The purpose of a Visual Question Answering (VQA) system is to answer questions related to a given picture. VQA systems have various applications, such as automatically generating picture descriptions and conducting searches using pictures as queries. This paper presents a VQA system designed specifically to provide automated assistance to blind individuals. The research utilizes the VizWiz dataset, the largest VQA dataset consisting of pictures taken by blind or visually impaired people. Pictures are processed by a Faster R-CNN model, which extracts regions of interest relevant to the posed questions. The output is a feature vector for each region. The questions are fed into a BERT model, which generates a single feature vector representing the entire question. The outputs from both models are combined using an attention layer. The output component of the system is a multi-label classifier that produces a list of probable answers. The system underperforms in comparison to the state-of-the-art. This can be attributed to the lack of fine-tuning for each individual component of the system, as well as shorter training times due to hardware limitations. Through error analysis, it was determined that the system tends to make more mistakes with blurry photos and images containing text. This represents potential directions for further research.