A Reinforced Passage Interactive Retrieval Framework Incorporating Implicit Knowledge for KB-VQA

Knowledge-Based Visual Question Answering (KB-VQA) requires integrating external knowledge to obtain answers. Recent research has attempted to acquire knowledge from external knowledge bases or large models. However, obtaining necessary knowledge based solely on image and question information is challenging and can be limited by retriever performance. This article proposes a novel retrieval strategy, namely implicit knowledge-driven explicit knowledge retrieval, which integrates the implicit knowledge in large models as auxiliary information with existing textual and visual information. This allows the retriever to locate external knowledge more accurately without overly relying on large models. To improve hit rate while maintaining retrieval efficiency, this paper proposes a passage self-interaction method. By allowing interaction between knowledge passage vectors and lightweight knowledge passage vectors, the dual-encoder can simulate the effect of a cross-encoder in a simple manner. Leveraging the advantages of the dual-encoder architecture, this method does not increase computational complexity during model inference. Experimental results demonstrate that this approach outperforms existing methods on the OK-VQA dataset.