In recent years, the concept of Explainable AI (XAI) has gained renewed interest as a significant research topic that dates back to the 1970s, largely due to the rapid progress in machine learning and the emergence of deep learning models. The main purpose of XAI is to explain the internal mechanism of models, often referred to as “Black Box”, and to give explanation regarding the reasons behind their predictions. A variety of methods, such as LIME, SHAP, and TCAV, have been proposed to explain either specific predictions or the overall behavior of these models. However, even with their impressive effectiveness and widespread adoption across different fields, these techniques struggle to propose a user-centered explanation. To handle this issue, researchers have considered ontology as a promoting solution to enrich XAI techniques due to its ability to provide more contextual and structured information about a given domain. Therefore, the main aim of this paper is to review and analyze these studies while presenting our proposition of this amalgamation in the area of Information Extraction (IE).