Stochastic optimization algorithms, particularly stochastic policy gradient (SPG), report significant success in reinforcement learning (RL). Nevertheless, up to now, how to speedily acquire an optimal solution for RL is still a challenge. To tackle this issue, this work develops a fast SPG algorithm from the perspective of utilizing a momentum, coined SPG-NM. Specifically, in SPG-NM, a novel type of negative momentum (NM) technique is applied to the classical SPG algorithm. Different from the existing NM techniques, we have adopted a few hyper-parameters in our SPG-NM algorithm. Moreover, the computational complexity is nearly the same as the modern SPG-type algorithms, e.g., accelerated policy gradient (APG), which equips SPG with Nesterov’s accelerated gradient (NAG). We evaluate the resulting algorithm on a classical task, the Markov decision process (MDP). Numerical results in different tasks demonstrate a faster convergence rate of the resulting algorithm by comparing state-of-the-art algorithms, which confirms the positive impact of NM in accelerating SPG for RL. Also, numerical experiments under different settings confirm the robustness of our SPG-NM algorithm for certain crucial hyper-parameters, which make the user feel free in practice.