Artificial intelligence (AI) mimics human intelligence by using machines and systems to solve everyday problems. Intelligence technologies such as machine learning and deep learning use algorithms to make accurate predictions of the future that do not require human intervention but can use cumulative model complexity and opaque black box models to achieve them. The phrase “interpretable artificial intelligence” (explainable artificial intelligence (XAI)) describes AI that can give human users interpretation of their decisions or predictions. XAI aims to drive transparency, trust, and accountability of AI systems high, especially when applied to health, economics, or security—occurs in stake applications. AI has become a mainstay in medical research because it has shown incredible success and promising results in various fields in the last 10 years. AI capability has been greatly enhanced by high data, computing efficient development, and creative designs. A comprehensive understanding of the AI industry is urgently needed, especially in key areas that are rapidly transforming research and improving personalized treatment. However, these systems often exhibit greater unpredictability and complexity, making it difficult to understand the options quickly. Physicians and patients in the medical field, where decision-making is highly influential, must fully trust AI systems. This requires a clear definition of the assumptions underlying the plans. In addition to building trust, this openness makes it easier to identify biases and flaws in the system. Recent years have seen the rise of semantic AI (XAI) as an important area of study, aimed at creating methods to improve AI systems semantics by providing decision-making insights. Definitions, classifications, standard procedures, techniques, advantages, barriers, and examples that characterize the position of XAI in medical imaging today are all discussed in the text. Saliency-based XAI techniques, which provide interpretation based directly on input data (such as images), are particularly emphasized in this paper. This is especially important for medical imaging as it can help to explain AI-generated semantics results afterward in this crucial area.