This paper proposes an explainability framework that integrates SHAP with LLM-generated knowledge to produce faithful, context-aware explanations for clinical decision support. Although machine learning models offer strong predictive performance, their black-box nature limits clinical trust. SHAP identifies influential features but lacks interpretive clarity, while LLMs can generate natural language explanations but often produce vague or unfaithful summaries. We introduce a two-stage pipeline utilizing SHAP values to guide the generation of relevant “LLM Reasons” and “Extensive Knowledge” summaries tailored for clinical interpretation. We applied the framework to clinical prediction tasks across three distinct datasets: MIMIC (emergency admissions), Neuroblastoma (cancer risk), and Elderly Care (elderly patrol needs). To evaluate explanation quality, we measured two metrics: Faithfulness Score, as the Jaccard overlap between top-k SHAP features and explanation content; and Novelty Score, as the cosine distance between explanation embeddings and a reference corpus. LLM explanations guided by SHAP showed the greatest benefit in behavior-driven tasks, with high novelty and alignment to model reasoning. Structured domains saw limited added value, as key features were already explicit. In semi-structured EHR, enriched prompts improved faithfulness and interpretability. Our LLM-based explanations, evaluated through Faithfulness and Novelty scores, provide clinically meaningful insights that SHAP alone cannot capture. This framework offers a scalable, domain-aware approach to improve explainability in clinical AI systems.