Model-agnostic explanation methods provide importance scores per feature by analyzing a model’s responses to perturbed versions of the sample to be explained. The explanation’s quality therefore hinges on the made perturbations and, most importantly, suffers if these lead to out-of-distribution samples. Unfortunately, this is the case for the popular LIME explanation method. In this paper, we thus introduce POMELO, an extension to LIME leveraging generative AI for full-input, in-distribution sampling. We define key properties of such samplers: distribution alignment, diversity, and locality. Based on these, we discuss different neural samplers based on normalizing flows and diffusion models. Our results demonstrate that neural samplers outperform traditional perturbation strategies and yield explanations that are better aligned with human intuition. Supplementary material to our paper is available at https://intellisec.de/research/pomelo .