Scene text visual question answering by using YOLO and STN

Extracting text from an image using a Visual Question Answering (VQA) system is an application at the intersection of computer vision and Natural Language Processing (NLP) to help blind people better view and comprehend textual information within the image. A VQA system takes a natural language question and an image as its input and then targets different areas of the image to extract an answer from the text within the image. To detect and recognize text in the image, the Look-Read-Reason-Answer (LoRRA), DCD_ZJU and Multimodal Multi-Copy Mesh methods, classify image features by using an Optical-Character-Recognition (OCR) module. Due to the curvature, distortion, background noise, and different fonts, understanding the text in natural scenes is a challenge as OCR techniques are mainly designed to detect textual features from within clean image files. In this paper, we introduce a novel architecture that rectifies perspective and curved text in an image using Thin-Plate-Spline (TPS) transformations and detects objects through the utilization of You-Only-Look-Once networks to extract answers. The results of our experiments show that our proposed VQA method reaches higher accuracy levels on the OpenImage dataset as compared with the existing methods.