Email and URL phishing attacks continue to pose significant threats to cyber security. Traditional detection techniques, such as heuristic-based and signature-based methods, often struggle with the dynamic nature of phishing tactics and the increasing use of adversarial techniques to evade detection. Machine learning (ML) and deep learning (DL) approaches, while promising, require extensive labelled datasets and computational resources, and can be vulnerable to adversarial attacks. The need for more robust, explainable, and adaptive phishing detection systems that can accurately identify phishing attempts across diverse platforms and communication channels is critical. In this paper, we propose a multimodal phishing detection system that analyses both email content and URL features. The system uses advanced pre-processing methods and ML models tailored for each phishing type and employs late fusion technique to integrate the outputs of the independently trained models. To make the system’s decisions more transparent, we incorporate explainable AI techniques, including Shapley Additive explanations (SHAP), Local Interpretable Model-agnostic Explanations (LIME) and Feature Importance (FI) to highlight the key features influencing the detection decision. Additionally, detected threats are processed by a Large Language Model (LLM) to provide context-aware defensive recommendations based on the MITRE D3FEND framework. Our extensive experiments, using 5-fold cross-validation, show high accuracy ( \(\approx \) 98% for emails and \(\approx \) 97% for URLs) and over 97% precision and recall, highlighting the effectiveness of our approach. This paper demonstrates that combining multimodal data and explainable AI enhances the robustness of phishing detection systems.