The rapid advancement of AI makes it ever more important to understand its decision-making—especially in safety-critical areas such as autonomous navigation. This PhD project investigates the self-explainability of autonomous systems using a self-driving ferry. It aims to develop a framework that enables the system to justify its decisions in a transparent manner. Using Explainable AI, Explainable Reinforcement Learning, Inverse Reinforcement Learning and Causal Modeling, the ferry’s expected behavior will be modeled, anomalies detected, causal relationships derived and self-explanations generated. This work could strengthen trust in autonomous systems and contribute to the general explainability of AI by opening the “black box” of its decision-making.