Deep reinforcement learning (DRL) has shown promising results in optimizing traffic signal control, with the Adam optimizer frequently delivering strong performance in training deep Q-networks (DQNs). However, the question remains: can alternative optimization algorithms outperform Adam by better escaping local minima and achieving superior global solutions? In this work, we present a comprehensive comparative study of the Adam optimizer and the classical Particle Swarm Optimization (PSO) algorithm for training DQN agents in a simulated four-way traffic intersection environment. Our simulation models real-world traffic dynamics using a cell-based approach and incorporates a reward function that balances queue minimization, throughput maximization, and penalization of frequent signal switching. We detail the mathematical modeling, code structure, and rationale behind every design choice, ensuring transparency and reproducibility. Experimental results demonstrate that while Adam achieves higher average rewards and lower queue lengths with greater computational efficiency, PSO offers more consistent training loss but lags in overall performance. This study provides valuable insights for researchers and practitioners seeking to select or design optimizers for DRL-based traffic management systems.