Mathematical equations describe fundamental laws across various disciplines, yet discovering concise and effective mathematical expressions from data remains a challenging task. Traditional symbolic regression methods often overlook domain-specific prior knowledge that scientists rely on, while large language model (LLM)-driven symbolic regression approaches can effectively leverage it. However, existing LLM-driven symbolic regression methods typically require substantial computational resources to generate equations while still suffering from low efficiency in producing high-quality expressions due to the lack of carefully designed prompts. To address this issue, we propose LLM-Guided Genetic Programming for Symbolic Regression (LLMGP-SR), a novel prompt-guided evolutionary search algorithm. LLMGP-SR integrates LLMs into the initialization, crossover, and mutation operations of genetic programming, achieving an organic integration of semantic generation and structural evolution of expressions. By leveraging an adaptive prompt strategy, LLMGP-SR constructs carefully designed prompts to guide LLMs in generating effective expressions. Experimental results demonstrate that LLMGP-SR significantly outperforms traditional genetic programming in symbolic regression problems across six standard benchmarks. The source code is available at https://github.com/chaoguo02/LLMGP-SR .