Deep reinforcement learning (DRL) has significantly advanced artificial agents in complex environments by integrating deep learning with reinforcement learning, demonstrating success in domains such as robotics, reinforcement learning from human feedback (RLHF), and game-playing. However, the alternating training and execution phases, particularly in on-policy methods, introduce substantial synchronization overhead, limiting efficiency. To address this challenge, we analyze the computational interplay between these phases and propose CoTraX, a novel framework that strategically overlaps training and execution to optimize resource utilization and accelerate training. Furthermore, we develop an adaptive control algorithm to mitigate potential adverse effects of overlapping. Extensive experiments demonstrate that CoTraX reduces training time by an average of \(9.89\%\) without compromising performance.