Several medical data processing applications often require the solution of inverse problems aiming at inferring original causal data starting from a set of measurements produced from those data. Existing deep learning solutions may suffer in generalizing to unknown measurement processes, therefore limiting their applicability to a reduced set of measurements. Generative AI can address conventional learning constraints by expanding the hypothesis space of all the possible physical models capable of producing the observed measurements. Indeed, generative models allow for excellently solving a larger set of medical inverse problems without prior knowledge of the process. However, the downside of not having a priori information available on real-world physical models is the lack of interpretability of such generative methods, which instead may be crucial in medical processes. To overcome such a shortage of information, explainability techniques for generative models should be developed. This paper introduces EXEGETE, an AI framework aiming at “interpreting” complex inverse problems in medical applications by developing new foundational generative AI methodologies. In particular, explainable generative AI methods will be developed for two specific and representative use cases in the medical domain: inverse problems in EEG signal analysis and inverse problems in breast cancer imaging.