Immune checkpoint inhibitors (ICIs) are a cornerstone of modern oncological treatments, particularly in the management of various cancers through immunotherapy. Despite their clinical success, ICIs are often associated with several immune-related adverse events (irAEs), among which pneumonitis is particularly significant due to its potential severity. Accurately identifying patients at high-risk of developing ICI-induced pneumonitis remains a critical challenge in lung cancer patient management. Early detection and precise differentiation are essential for timely and appropriate therapeutic interventions, which can significantly alter patient outcomes. We developed the PANDA (Pneumonitis ANomaly Detection using AttentionU-Net) model to address this challenge, leveraging advanced deep learning techniques to improve the early predicting of ICI-induced pneumonitis. Baseline CT scans from 348 cases (33 pneumonitis cases) patients undergoing ICI therapy were analyzed to train and validate the model. The PANDA model utilizes the Attention U-Net architecture, incorporating attention mechanisms to enhance feature extraction and anomaly detection capabilities. Data augmentation techniques, including brightness normalization and pixel shuffling, were applied to improve model robustness. The model was trained on normal cases using an autoencoder-based method with anomaly detection through mean squared error (MSE) distribution, followed by testing on pneumonitis cases. The PANDA model demonstrated superior performance, achieving a precision of 0.76, sensitivity of 0.79, specificity of 0.79, F1-score of 0.78, AUC of 0.85 and a Precision-Recall AUC of 0.82. These results significantly outperform traditional models, including clinical and radiomics approaches. The clinical model, for instance, achieved a precision of 0.75, sensitivity of 0.67, specificity of 0.73, F1-score of 0.76, AUC of 0.69 and a precision-recall AUC of 0.76. The classical radiomics model showed improvements over the clinical model, with a precision of 0.81, sensitivity of 0.72, specificity of 0.80, F1-score of 0.76, AUC of 0.70 and a precision-recall AUC of 0.79, but still fell short of the PANDA model’s performance. These comparisons emphasize the enhanced predictive capacity of the deep learning approach, significantly outperforming traditional models.