Retrieving relevant information from long videos remains a significant challenge due to high computational costs, semantic redundancy, and the need for temporal reasoning. We propose LightVideoRAG, a lightweight retrieval framework tailored for long-video question answering. LightVideoRAG combines adaptive frame sampling, which filters out redundant frames while preserving key semantic content, with context-aware retrieval modules that integrate both local neighborhood signals and global temporal information. This design enables efficient temporal grounding without processing entire video sequences. Unlike existing methods that rely on dense captioning or proprietary APIs, our system operates entirely on a locally deployed Vision-Language Model (VLM), ensuring strong data privacy and low latency. Evaluations on the LongVideoBench and Video-MME benchmarks show that LightVideoRAG achieves substantial gains in QA accuracy while requiring only a fraction of the computational resources, outperforming the base model and approaching the performance of larger size models. This demonstrates its potential as a scalable and accessible solution for efficient video understanding in resource-constrained environments. Our code is available at https://github.com/linshys/lightvideoRAG .