Integrating multi-modal Large Language Models (LLMs) into the healthcare infrastructure has the potential to transform clinical workflow. Through effective processing and reasoning from unstructured text and complex medical imagery, these models have the potential to significantly improve documentation efficiency, diagnostic accuracy, and personalized patient communication. But their unprecedented scale and immanently generative, emergent patterns of behavior place a discomfiting opacity, as an inherent barrier to their safe, ethical, and dependable deployment in sensitive clinical applications. To counter, the new field of Explainable LLMs (XLLMs) seeks to unveil these potent models by making them transparent to their decision-making. Above all, the rapid advance in the formulation of XLLM explanation techniques outpaced the design of scientifically valid, standardized, context-sensitive techniques for their verification. Creating an explanation is only a start; its actual value depends on its verifiable fidelity, clinical plausibility, factual correctness, and practical utility. This chapter provides a detailed and organized presentation of the complex problems involved in assessing explainability for LLMs in particular within the healthcare setting. We start by defining the special challenges that these models pose, and then rigorously analyze the evaluation domain into two main, complementary paradigms. First, a comprehensive review of technical metrics is presented, extending root notions such as fidelity and robustness to the multimodal environment, and introducing new metrics to estimate factual grounding. Second, a multi-level human-oriented evaluation framework is presented, describing methodologies for measuring the impact of XLLM explanations on clinical plausibility, practitioner trust calibration, and ultimately, decision-making effectiveness. Finally, we define the demand for comparative benchmarking. Comparative measurement of XLLM evaluation methodologies against the state of the art in current science on established, publicly accessible benchmarks help to render claims of improvement, replicable, and appropriately situate it in the broader scientific framework. This promotes continuous methodological advancement and standardization of assessment practice towards facilitating responsible and effective LLM implementation in healthcare.