Cross-modal visual information has been demonstrated to enhance the performance of unimodal text tasks. However, efficiently acquiring and utilizing this cross-modal visual information remains a significant challenge. To address these issues, we introduce a novel model named Multimodal Model with Cross-modal Retrieval-Augmented and Visual Region Perception (MM-CARP), which leverages additional visual knowledge retrieved as supplementary information to enhance text-based Question-Answering (QA) tasks. MM-CARP operates in two stages: retrieval stage and generate stage, which are achieved by the Cross-modal Retrieval-Augmented (CRA) module and the Cross-Modal Generation (CMG) module, respectively. Specifically, the CRA retrieves multiple visual knowledge candidates based on the input question, and then encoding the text and visual knowledge into a shared multimodal semantic space. The optimal solution is selected as supplementary information by calculating the distance between text and visual knowledge embeddings. The CMG performs reasoning based on the textual input and external visual knowledge. Notably, to enable the MM-CARP model to effectively utilize visual information during the generation stage, we have introduced a Visual Region Perception (VRP) module. This module captures regional information through an end-to-end approach, thereby augmenting the model’s focus on visual region content. Experimental results on three question-answering datasets validate the efficacy of the MM-CARP model in enhancing QA capabilities through the use of cross-modal data.