Hybrid CNN and Low-Complexity Transformer Network with Attention-Based Feature Fusion for Predicting Lung Cancer Tumor After Neoadjuvant Chemoimmunotherapy

Neoadjuvant chemoimmunotherapy is an effective treatment for lung cancer. Accurate prediction of lung cancer tumor after neoadjuvant chemoimmunotherapy from Computed tomography (CT) scans is of significance as it can enable clinicians to quickly select patients who will obtain good outcomes after treatments and evaluate patients’ prognosis in advance. Therefore, this study proposes a new hybrid Convolutional Neural Network (CNN) and low-complexity Transformer network with attention-based feature fusion to achieve lung cancer tumor prediction after neoadjuvant chemoimmunotherapy on CT scans. In each stage, we first use channel split to reduce the parameters, then we design both CNN branch and low-complexity Transformer branch to learn the effective local and global features. To reduce the computational complexity of the Transformer, we introduce a novel low-complexity self-attention computation method via feature aggregation and element-wise multiplication. Additionally, to effectively fuse CNN and Transformer features, we design an attention-based feature fusion module with coarse fusion, attention computing and fine fusion. The proposed method is evaluated on a private lung cancer dataset including CT scans of 232 patients undergone neoadjuvant chemoimmunotherapy. Experimental results demonstrate that our proposed method achieves the highest Dice of 47.04% with the fewest parameters of 2.91M and floating point operations (FLOPs) of 52.95G, outperforming several state-of-the-art methods. Diameter analysis shows the predicted diameters of lung cancer tumors are promising to provide support information for treatment planning.