This paper proposes a solution for the challenging task of automatically solving (JEE) Mains and Advanced questions in Physics, Chemistry, and Mathematics (one of the toughest exams globally). No work has been done in this field yet. Leveraging a dataset comprising questions from previous years’ exams, we investigate the capacity of LLMs (Naveed et al. A Comprehensive Overview of Large Language Models (Version 8), 2023) to understand and solve complex academic problems, utilizing advanced methodologies like Retrieval-Augmented Generation (RAG) (Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Version 4), 2020). Our comprehensive evaluation, utilizing a dataset of 610 problems, demonstrates that the integration of RAG significantly improves the problem-solving capabilities of LLMs. Specifically, the study presents a marked improvement in accuracy, with the advanced method achieving a total accuracy of 0.763 on JEEBench (Arora et al. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models, 2023) (for JEE Advanced) and 0.854 on JEEMB (for JEE Mains), compared to the baseline accuracy of 0.218 and 0.521, respectively. These results underscore the potential of LLMs, augmented by additional relevant context.