Pneumonia is a life-threatening respiratory infection. For its binary classification, we propose a lightweight and interpretable baseline for low-resource settings. Our approach integrates benchmarking with Transfer Learning and Vision Transformer (ViT) families, while incorporating robustness and deployment-oriented metrics. Specifically, we develop a computer-aided diagnosis (CAD) system comprising a Convolutional Neural Network (CNN) with a Squeeze-and-Excitation (SE) Attention Layer. We further rely on DenseNet121 and ResNet50 and training from scratch for resource efficiency. We use CLAHE (Contrast Limited Adaptive Histogram Equalization) for image enhancement with augmentation and weighted loss, to tackle class imbalance and to boost training robustness. Evaluated on the Kaggle Pneumonia Dataset, our model achieves 0.9750 accuracy and 0.9798 F1-score with low loss and faster training times. Lastly, for generalization purposes, we used NIH Chest X-ray 14 Dataset, RSNA Pneumonia Dataset, and CheXpert Dataset. Furthermore, Grad-CAM++ and SHAP provide visual and feature-level insights, aligning with clinical needs for interpretable diagnostics. The results demonstrate that the proposed baseline achieves strong performance and reliability, making it a reasonable practical choice for real-world resource-constrained settings.