In the literature on Industry 4.0 and 5.0, the applications of machine learning algorithms for process optimization are widely described; however, few studies consider the perspective of the user who makes operational decisions based on these algorithms. Explainability of artificial intelligence models is understood as the system’s ability to present, in a human-understandable way, the premises, mechanisms, and limitations underlying the generated predictions or decision recommendations. In the context of manufacturing management, explainability serves as a bridge between advanced analytical methods and managerial practice, enabling informed use of model outputs under conditions of process variability, time pressure, and accountability for decision outcomes. This article proposes a conceptual model describing the relationships between explanation quality, decision-maker trust, prediction utility, and decision effectiveness. It also presents dimensions for evaluating explainability specific to the production environment and discusses implications for management, auditing, and organizational safety. The work provides a theoretical foundation for further research on XAI metrics and proposes a framework for implementing transparent AI systems in manufacturing enterprises.