The black-box nature of deep learning still prevents its widespread clinical use due to the high risk of hidden biases and prediction errors. Over the last decade, various explanation methods have been proposed to reveal the latent mechanisms of neural networks and support their decisions. However, interpreting the explanations themselves can be challenging, and there is still little consensus on how to evaluate the quality of explanations. To investigate the fidelity of explanations provided by prominent feature attribution methods for Convolutional Neural Networks in Alzheimer’s Disease (AD) detection, this paper applies relevance-guided perturbation to the Magnetic Resonance Imaging (MRI) input images. According to the fidelity metric, the AD class probability showed the steepest decline when the perturbation was guided by Integrated Gradients or DeepLift. We conclude by highlighting the role of the reference image in feature attribution with regard to AD detection from MRI images. The source code for the experiments is publicly available on GitHub at https://github.com/bckrlab/ad-fidelity .