This paper explores the potential benefits and challenges of implementing Artificial Intelligence (AI) technologies in critical infrastructure systems, particularly Explainable Artificial Intelligence (XAI). As the shortage of skilled workers continues to grow, automating monitoring and control duties in critical infrastructures becomes increasingly necessary. However, the lack of transparency, trust, and explainability in traditional AI approaches creates hesitation among stakeholders. This paper evaluates the regulatory requirements for AI systems in critical infrastructures, analyzes how XAI can enhance resilience in these systems, and explores the potential design of an XAI-based autonomous system for critical infrastructure. Regulations from the European Union, Canada, and the United States are analyzed to evaluate the overall regulatory framework for AI applications in critical infrastructure. It also discusses how AI could optimize the performance of different critical infrastructure systems by predicting potential failures, reducing downtime, and extending the lifespan of infrastructure components. However, the lack of transparency in traditional AI techniques becomes a significant concern in fields where understanding the rationale behind decisions is crucial for ethical, legal, and practical reasons. The paper concludes by discussing how XAI can foster trust in AI systems through greater transparency and enhance the resilience of critical infrastructure systems.