LLC resonant converters possess application advantages in the field of AI chip power supply. Conventional model-based methods for tuning controller parameters suffer from poor dynamic performance and low engineering generality. To address this issue, this paper proposes a Frequency-Domain Anchored Reinforcement Learning method with Two-Stage Constraints for optimization. The method screens reference parameters through frequency-domain design to anchor the search center for the Reinforcement Learning Q-learning algorithm, sets up multi-index differentiated rewards to achieve parameter optimization, and adopts two-stage constraints to accurately focus on the effective parameter range, reducing redundant calculations and improving search efficiency. The optimization of specific performance metrics is achieved by allocating weights to parameters in the Reinforcement Learning reward function. Simulation verification shows that the optimized parameters significantly accelerate the response speed and disturbance recovery speed, enabling the new system to balance stability, dynamic performance, and anti-interference capability.