Multi-turn conversational AI systems using Retrieval Augmented Generation (RAG) often struggle with ambiguous user queries and inefficient retrieval from large document collections. Traditional chunk-level vector search introduces significant computational overhead, while missing critical context due to reliance on raw, under-specified queries. We propose ConvSelect-RAG, a novel three-stage framework that (1) enhances queries using conversation history, (2) pre-filters documents using metadata summaries before chunk-level retrieval, and (3) integrates both for context-aware response generation. Experiments on large conversational QA benchmarks demonstrate that ConvSelect-RAG reduces retrieval latency by 23.5%, improves response accuracy by 18.7%, and decreases overall computational overhead by 31.2% compared to existing RAG baselines. Our approach offers superior scalability for real-world applications by minimizing unnecessary searches and preserving contextual relevance, setting a new standard for efficient, accurate multi-turn conversational AI.