CBFA-Net: Cross-Branch Feature Alignment for Trustworthy Medical Image Classification

Trustworthy and interpretable artificial intelligence is increasingly important in computer-aided diagnosis, where clinicians require not only accurate predictions but also reliable and clinically meaningful decision support. In hybrid medical image classification, global contextual semantics and fine-grained local structural cues are often combined to improve performance; however, direct interaction between these heterogeneous representations can introduce distribution mismatch, feature interference, and unstable fusion. To address this issue, we propose CBFA-Net, a Cross-Branch Feature Alignment Network for medical image classification. CBFA-Net employs a Vision Transformer backbone, a CBAM-based global refinement branch, and a depthwise separable convolution-based local refinement branch. The proposed Cross-Branch Feature Alignment module enables compatibility-aware bidirectional interaction by aligning each source representation to the statistical space of the target branch before residual exchange. The aligned features are then combined through adaptive gated fusion for classification. Experiments on the Kvasir and ISIC 2018 datasets show that CBFA-Net achieves 98.91% and 93.56% accuracy, respectively, outperforming existing methods. Ablation studies confirm that the alignment module is the main contributor to performance gains, while Grad-CAM visualizations show that the model focuses on clinically meaningful regions. These results demonstrate that compatibility-aware feature alignment provides an effective and interpretable solution for trustworthy medical image classification.