Improving XAI Explanations for Clinical Decision-Making – Physicians’ Perspective on Local Explanations in Healthcare

Healthcare faces significant global challenges due to an aging population and the surge of chronic health conditions. Artificial Intelligence (AI) has emerged as a promising tool to address these issues, but the lack of transparency hampers clinicians’ trust. Explainable AI (XAI) is a method to explain the AI’s logic behind its predictions, thereby improving transparency. However, very little research has been done on XAI from the clinicians’ perspective. Our research aims to understand how clinicians want the XAI explanations of medical AI predictions to be presented. A total of 30 physicians from 9 medical specialties evaluated two of the most used XAI methods on AI predictions on patients: Local Interpretable Model-agnostic Explanations (LIME) and Diverse Counterfactual Explanations (DiCE). A mixed method approach was used with web-based questionnaires, and the results were analyzed with statistical and thematic methods. We found a significant disparity: the explanations generated by XAI methods often fail to align with physicians’ expectations and requirements for reducing uncertainty and providing enough depth of explanation in AI predictions. Moreover, there is a paradox where the areas rated most important by the physicians performed the poorest, whereas XAI excels in areas not rated equally essential. The physicians also highlighted additional areas that need to be addressed, including general upskilling of clinical staff in AI and XAI, ensuring the AI and XAI tools are integrated into the normal healthcare processes, and the ability to personalize the XAI explainability presentations.