Convolutional Neural Networks are extensively employed in critical domains such as computer vision, medical imaging, and autonomous systems. Enhancing model interpretability by providing users with concise and context-relevant explanations of CNN decision making such as visualizing feature maps or saliency regions, enables a deeper understanding of the model’s internal representations and inference process. The proposed work presents a deep learning framework integrating a ResNet-based U-Net architecture with a Fixation Point Generator (FPG) to perform classification and saliency aware reconstruction on the hand-written dataset. The model leverages transfer learning by employing a pre-trained ResNet-18 as the encoder backbone, enabling robust feature extraction. A custom decoder reconstructs input images while a classification head predicts digit labels. To enhance model interpretability, a Fixation Point Generator predicts spatial attention maps (saliency maps) from high-level global features, highlighting regions of interest that influence model decisions. This implementation aims to bridge the gap between classification performance and model explainability, offering insights into the model’s focus areas through learned attention. The model got an accuracy of 98.44 on the Malayalam handwritten dataset, 97.81 on English handwritten dataset, and 99.56 on the MNIST dataset.