Due to the strong reasoning capacities of large language models (LLMs), grammatical error correction (GEC) has significantly improved. However, the deficiency of external knowledge in GEC models may lead to a semantic bias in candidate corrections. To address this issue, we propose a Syntax-Aware Grammatical Error Correction (SAGEC) approach to select effective retrieval syntax pairs from LLMs, thus guiding our GEC models to obtain the best candidate corrections. First, we collect the syntax preference corpus based on the accuracy of candidate corrections generated by LLMs. Second, we exploit the Monte Carlo Tree Search (MCTS) method to benefit LLMs by probing useful retrieval syntax pairs from the syntax preference corpus. Third, yielded retrieval syntax pairs are leveraged to fine-tune LLMs, thus obtaining syntax-aware LLMs with strong syntax reasoning ability. Finally, we leverage syntax-aware LLMs to re-rank retrieval syntax pairs and integrate them into LLM-based GEC models. Experiments on several benchmark datasets show that our proposed SAGEC model outperforms all strong baselines, leading to state-of-the-art results on both English and Chinese datasets. Detailed analyses reveal that our SAGEC model can distinguish syntax knowledge effectively and alleviate the semantic basis in normal grammatical error correction. Our codes will be available at https://github.com/0cloud-hub/SAGEC .