<p>In a Visual Question Answering model, the interaction of image region and question word is crucial as integrating the information from these two modalities results into a combined representation which allows the model to understand the relationship between the image and question. Through bidirectional cross-attention based interaction, the visual content is aligned based on textual semantics and vice versa. However, focus on every image region and question token might also include irrelevant information. A soft-attention based cosine similarity modulator is proposed in this paper which dynamically compares the elements of the input texts and image, emphasizing relevant components in the respective input while deemphasizing less important ones. The modulated features are then used for cross-modal attention, where the refined query interacts with key for enhanced representation. The answer prediction stage has another soft-attention mechanism to integrate the feature embeddings from different feature extraction branches based on the attention score, thereby enhancing the model’s ability to generate accurate answers. To evaluate the performance of the model, a flexible domain VQA model with shared architecture is proposed. The model is evaluated on the following general, medical and remote sensing VQA datasets, i.e., Visual7W, VQAv2, VQA-MED 2019 and RSVQA-LR. The performance across these benchmarks demonstrates the model’s effectiveness in handling both domain-specific and general VQA tasks, highlighting its adaptability and robustness.</p>