Infrared (IR) and visible (VI) fusion aims to combine thermal radiation information in IR images with rich texture details in visible images. However, existing methods tend to ignore the inter-modal disparity information, thereby limiting fusion performance. In this paper, we propose a bimodal difference-aware attention network (BiDAFuse) for effective IR and VI image fusion. Our BiDAFuse consists of a difference-guided feature adjustment module (DGFA) and a bimodal attention fusion module (BIAF). The DGFA dynamically adjusts infrared and visible features using pixel-level disparity maps and a gating mechanism to enhance modality-specific information. The BIAF uses channel and spatial attention to adaptively weight the features, capturing long-distance dependencies and critical details. We evaluate the effectiveness of BiDAFuse through extensive experiments on two publicly available datasets (TNO and M3FD). Compared to seven state-of-the-art fusion methods, BiDAFuse demonstrates superior performance in terms of visual quality, quantitative metrics, and generalization capability.