In a rapidly evolving technology landscape, timely access to appropriate and relevant information, be it for business or personal use using LLMs (Large Language Models), is of paramount importance. But there is a flip side to all this, putting your data into these proprietary LLMs has given rise to the ever-present concern about privacy and data security, particularly when users effectively trade their data into popular online LLMs-based systems of questionable provenance. In this paper, we delve into the practical implementation and thorough evaluation of a system that combines retrieval and generation techniques to enhance natural language understanding and response generation. We utilize the Llama 2, Phi-3, Mistral, etc. models alongside other open-source tools to build our system. Our approach involves retrieving relevant documents, creating embeddings, and generating responses based on both the query and retrieved information. We evaluate our system using various metrics to assess its performance and effectiveness. Conclusion: This research addresses the privacy concerns by designing a context-aware retriever for information retrieval and interaction that is privacy-first. It uses locally stored open-source LLMs instead of depending on generic, opaque online LLM platforms. The model ensemble_rag achieved high scores in key dimensions, such as context precision, faithfulness, and answer correctness. The strong points of the basic_rag and pdr_rag models were relevancy of answers and recall of contexts.