Accurate trajectory tracking in lane-change maneuvers is crucial for autonomous driving. This study integrates reinforcement learning (RL) with a dual-loop PID controller to enhance performance. The outer loop controls position, while the inner loop manages orientation, with its gains (Kp, Ki, Kd) dynamically optimized via the PPO algorithm. Simulations comparing RL-enhanced PID (RL-PID) and conventional PID under varying gain settings confirm superior tracking accuracy and stability with RL-PID. Results highlight that adaptive gain tuning significantly improves control robustness and adaptability. This research advances intelligent vehicle control strategies for complex driving scenarios.