Current methods exhibit limited accuracy in detection and localization when facing diffusion model-based image tampering that are semantically coherent yet leave only faint traces. To meet this challenge, we introduce CABF-Net, a dual-branch framework for image tampering detection and localization. The image-level branch employs tampering-aware adaptation of CLIP model to leverage its powerful global semantic representations for robust tampering detection. The pixel-level branch uses VMamba with Multi-Forensic Signal Guidance (MFSG) strategy, enabling fine-grained and accurate localization of tampering regions. More importantly, we design a Cross-Attention Bidirectional Fusion (CABF) module to enable deep interaction and fusion between semantic and pixel-level features across the two branches. Moreover, we construct MAF-set, a large-scale and diverse dataset of diffusion model-based image tampering, to fill the gap of limited quantity and diversity in existing datasets. Extensive experiments on public benchmarks and on our proposed dataset MAF-set demonstrate that CABF-Net significantly outperforms current state-of-the-art approaches in both detection accuracy and localization precision. The code and MAF-set are available at https://github.com/Samarium-Z/CABF-Net .