<p>Transparency in the Artificial Intelligence (AI) models represents the future potential that the research community seeks, prompting a strong call for eXplainable Artificial Intelligence (XAI) and significantly transforming the dynamics of trust and risk evaluation. Learning models operate as black-boxes, where users input a sequence of instances for labeling without gaining insight into the model’s learning or behavior. This can be handled by incorporating an explanation along with the prediction, thus evolving from a black-box to a white-box model, making it more transparent and interpretable. XAI has found successful applications across diverse domains, especially in medical and banking. However, its exploration within Optical Character Recognition (OCR) remains relatively unexplored. This paper presents an explainable model that provides a visual understanding of the predictions made by a CNN model to recognize the handwritten Assamese digits. This in turn ensures trust in the black-box model predictions thus making it transparent. A Gradient Class Activation Mapping (Grad-CAM) is incorporated with a CNN model that gives visual explanations against the predictions made. The visual explanations are justified by human evaluation further, where the explanations are validated. Through these visual explanations, users can make judgments regarding their trust in the model and determine the fidelity of its predictions. The visual explanations generated by the Grad-CAM in the present work demonstrate that the model’s predictions align with human understanding, thereby establishing the faithfulness and trustworthiness of the model in predicting the digits.</p>