Reinforcement Learning (RL) is becoming popular for two-legged robots to learn and improve, through trial and error to adjust their actions based on feedback. Deep RL, combining RL with deep learning, handles high-dimensional states and action spaces in robotics. Two important Deep RL algorithms are Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3). Our proposed new algorithm that improves TD3 involves to maximize cumulative rewards by interacting with the environment. The robot learns a policy through exploration and exploitation. This paper investigates the scenario of robots continuously walk without falling. Our experiments show that the two legged robot can automatously adapt to the environment by themselves, without human solving the problems for them. DDPG shows promise but suffers from instability and hyperparameter sensitivity. Our improved TD3 mitigates DDPG’s overestimation bias, improving stability and performance. This study also evaluates stability, convergence, and computational efficiency of DDPG and TD3.