Machine-learning models have grown increasingly complex, making them correspondingly more difficult to interpret. The field of explainable AI (XAI) has emerged to address this challenge by developing techniques to explain and interpret these models. We believe there is considerable unexplored potential for crossover between evolutionary computation (EC) and XAI, which this chapter aims to highlight. We present a problem-focussed taxonomy of XAI techniques and a brief survey of notable methods for explaining machine learning and particularly deep learning models, with an emphasis on those which incorporate EC. We also discuss EC-based strategies for building more interpretable neural networks, such as neuroevolution. This chapter seeks to highlight the untapped potential of EC in improving transparency in machine learning and to inspire future research in this direction.