The deployment of large language models (LLMs) in health question-answering (QA) systems presents unique challenges, particularly on scalability, inference time, and hallucination mitigation. While Retrieval-Augmented Generation (RAG) methods have demonstrated promise, current implementations do not fully address the complexities of medical content, which is both keyword-dense and semantically intricate. In this work, we present LADDER, a question decomposition framework tailored for practical, real-world medical QA applications. By creating a reduced knowledge base (RKB) and proposing a novel latent self-attention mechanism, LADDER improves the retriever model’s ability to extract more relevant information. We demonstrate this on a medical setup, achieving a 2.5% increase in retrieval performance and a 6–8% improvement in generative prediction performance over existing MedRAG benchmarks. Additionally, inference time is reduced by 47%, enhancing system responsiveness in real-world conditions. This approach not only meets the performance demands of healthcare systems but also ensures robustness against hallucinations, making it viable for real-world deployment under computational and operational constraints.