Explainable Artificial Intelligence (xAI) techniques are nowadays widely accepted as one of the paths towards addressing the interpretability and transparency issues of using black box models. Such techniques may allow to understand, to a certain extent, how or why a model produced a certain output, which may even help identify problems with the model or the data. As in many other domains, the use of xAI techniques in the context of manufacturing is seen as fundamental towards understanding model outputs, supporting informed decision-making, or enabling more human-centric approaches. In this paper, we specifically look at LIME, one of the most widely used approaches to xAI, and at how it needs to be adapted to the manufacturing context. Specifically, we show how the image permutations introduced by LIME might deceive the underlying model and generate poor explanations, and propose a methodology to address this issue. The specific use-case is on defect detection in the textile manufacturing industry.