Question-Guided Hybrid Learning and Knowledge Embedding for Visual Question-Answering

Visual question-answering is an important application of the fusion of vision and language in multimodal learning. Its basic task is to understand the input 2D image or 3D point cloud and answer text questions based on it. At present, the development of visual question and answering technology is facing some problems. In some fields, it is difficult to obtain accurate answers by inferring only the information contained in images and problems. The data imbalance in practical application scenarios also limits the ability of models to handle rare or complex problems. In this article, we propose an image question-answering model based on problem-guided hybrid learning and knowledge embedding. This model queries relevant knowledge in the knowledge graph and integrates the queried knowledge with the problem text to form new textual information. During the training process, tuples of similar problem types (v, q, a) are mixed to generate new data samples. Then, feature extraction and cross modal fusion are performed on the new samples, and the samples are fed into the answer prediction network to obtain the answers. The optimization model narrowed the gap between predicted answers and mixed answers, ultimately enabling the model to generate more accurate answers. A series of comparative experiments conducted on the OKVQA dataset and the SLAKE dataset has verified that the model can effectively improve the accuracy of image question-answering.