Visual Question Answering models have been shown to suffer from language bias, where the model relies too much on surface correlations or specific language patterns between the questions and the answers when answering a question, and ignores image information. Previous works to reduce this bias have used question-only models or data augmentation, but they have overlooked the effective utilization of correlated information between text and vision. In this paper, we propose a solution to address these challenges by introducing a novel Bidirectional Vision-Language Guidance approach called BIVL-Net. BIVL-Net redirects the focus of the two modalities by introducing image categories as additional information and treating them as inter-modal association objects. Our proposed framework comprises two key components: the Textual Information Enhancement Module (TE) and the Question-Guided Visual Enhancement Module (VE). In the TE module, we adjust the model’s focus on textual features to mitigate biases introduced by the textual modality. VE module optimizes the final image feature representations by using the objects obtained from the text side as prompts to guide the visual side to focus on the objects related to the questions. Finally, to improve the model’s discrimination of question types, we propose a relational optimization loss \(l_{rl}\) that is effective for both in-distribution(ID) and out-of-distribution (OOD) data. We demonstrate the effectiveness of the proposed method on the OOD dataset VQA-CP v2 and the ID dataset VQA v2. In addition, this paper is also the most robust in terms of differences between ID and OOD data.