This study evaluates the sufficiency of Explainable AI (XAI) techniques in clarifying the decisions of AI models, with the goal of developing a framework that provides reliable and comprehensible explanations. The framework is designed to ensure that medical practitioners can trust and effectively apply these insights in clinical practice. To validate the framework, a VGG16-based model trained for glaucoma prediction using retinal fundus images is employed. The model’s predictions are processed through the framework to generate explanations using various XAI techniques. These explanations are assessed to determine their ability to sufficiently elucidate the relationship between input features and predicted outcomes. The study focuses on post-hoc explanation methods to ensure accessibility for non-expert users, such as medical practitioners, who require clear and interpretable insights. This research is limited to publicly available medical datasets and pre-trained models, which may constrain the generalizability of the findings to diverse clinical environments or proprietary datasets. Broader validation with varied datasets is recommended to enhance the applicability of the proposed framework in real-world medical settings.