Graph-based methods have shown strong performance in high-dimensional approximate nearest neighbor (ANN) search but often face bottlenecks in meeting the stringent requirements of low latency and high recall. Recent hybrid approaches, such as SymphonyQG and HNSW, integrate graph structures with quantization techniques to balance accuracy and efficiency. However, these methods still suffer from excessive memory consumption and performance instability, particularly during hash table rehashing. In this work, we propose LiteQG, a lightweight and efficient ANN search framework designed to address these limitations. Specifically, LiteQG introduces two key innovations: (1) replacing traditional quantization with 8-bit scalar quantization (SQ8) to significantly reduce memory overhead, and (2) designing a novel lazy-cleanup set that eliminates the overhead of conventional hash sets, leading to more stable query performance. Extensive experiments on the SIFT dataset demonstrate that LiteQG achieves a query processing rate of 20,332 queries per second (QPS) while maintaining a 99% recall rate. Compared to state-of-the-art baselines, LiteQG reduces memory usage by 36.7% and achieves approximately 6× performance improvement in query efficiency.