Explainable artificial intelligence (XAI) is an important research area for understanding and accessing AI systems. It wants to shed light on the rationale and reasoning behind complicated machine learning models’ decision-making by providing insights into this rapidly evolving topic. This study stresses XAI’s use in identifying bone fractures using an X-ray imaging dataset (containing fractured and non-fractured photographs) and classifying photos using a 5-class floral image dataset. Explainable AI (XAI) clarifies the ‘black box’ nature of AI systems, fostering confidence and widespread adoption. The project aims to improve CNN model interpretability using XAI. This study used the CNN architecture to analyse floral image collections and bone fracture X-rays. With 88% and 98% training and validation accuracies on the bone fracture dataset, the CNN models performed well. The CNN model trained and validated the five-class flower image dataset with 92% and 88% accuracy. To further analyse the CNN model’s conclusions, the local interpretable model-agnostic explanations (LIME) XAI technique was used. LIME emphasises input picture components that significantly influenced the model’s decision to produce locally precise explanations for specific predictions. CNN model predictions with image analyst and physician inputs aim to increase confidence, utility and real-world application utilising LIME. This work uses the LIME technique to balance algorithm obscurity with interpretable healthcare and image analysis decision-making.