Towards an Explainability Agent: Leveraging LLMs to Interpret LIME Outputs
摘要
In clinical decision-making, the adoption of machine learning models requires not only high predictive performance but also transparent, trustworthy explanations. This work presents a hybrid explainability framework that combines Local Interpretable Model-agnostic Explanations (LIME) with a Large Language Model (LLM) to generate natural language explanations of classification outputs in healthcare applications. While LIME provides local feature attributions for individual predictions, these are often difficult to interpret by non-technical clinical staff. Our system leverages an LLM to translate LIME outputs into fluent, domain-aware explanations that align with the reasoning needs of healthcare professionals. We evaluate the method on two medical datasets involving patient risk classification tasks and obtain promising results in terms of interpretability and consistency with the model’s decision logic. However, the current evaluation lacks validation by external healthcare professionals, which we identify as an essential next step. Despite this, our approach represents a strong foundation for the development of adaptive explainability agents in clinical contexts. We discuss its potential impact on future decision support systems and propose directions for evolving toward interactive, trustworthy, and user-aligned AI tools in medicine.