Advancing Web-Based Visual Question Answering with Efficient Image-Text Alignment

It is important to examine multi-hop web-based VQA datasets such as WebQA because such benchmarks require information retrieval on both image and text sources, better aggregation and summary of knowledge, and higher reasoning ability on open-domain questions. By implementing and analyzing the baseline models on WebQA, we found that the image resources are not fully extracted and understood and the model is heavily dependent on text. In addition, the large pre-trained VLP model is very time and memory consuming. Therefore, we want to investigate alternatives which align text and image via image-text matching loss and multimodal cross-attention module. We also aim to reduce the size of the model and improve training speed by applying some lighter models via detector-free visual encoders and knowledge distillation.