Symbolic regression (SR) is a technique that explicitly captures the hidden patterns in data by expressing them as mathematical formulas. Recently neural network-based (NN) approaches have gained increasing attention for this NP-hard problem. For instance, Neural Encoded Expression Programming (NEEP) utilizes the sequential modeling capabilities of Recurrent Neural Networks (RNNs) to encode and construct mathematical expressions. While NEEP employs evolutionary algorithms (EAs) for global optimization, it lacks gradient guidance for fine-grained search. Additionally, its input-free encoder suffers from limited context awareness and lacks dynamic feedback. To overcome these challenges, this paper proposes NEEP-RLAO, a novel framework integrating EA with reinforcement learning (RL) to solve SR problems. RL enhances EA by performing gradient-based optimization on high-quality individuals identified during exploration, which are then reintegrated to maintain diversity and improve solution quality. Additionally, a new autoregressive encoder is designed to improve expression continuity and adaptability to complex problems. Experimental results on various challenging benchmark problems demonstrate NEEP-RLAO has better performance compared to several representative methods on symbolic regression.