Reinforcement learning (RL) enables agents to learn optimal decision-making policies by interacting with an environment, guided by reward signals within a Markov Decision Process (MDP) framework. This paper investigates the mechanisms of forward and backward reward propagation in RL, analyzing their impact on state-value updates and policy for balancing immediate reward and delayed reward. Using a grid-world environment, we implement a Q-learning agent that leverages both forward (experience accumulation) and backward (reward adjustment) propagation to estimate expected rewards for choosing optimal actions. The study demonstrates how backward propagation accelerates convergence in reward estimation and explores the influence of environmental constraints, such as obstacles and state-dependent actions, on learning dynamics. Experimental results, visualized through state-value heatmaps, provide insights into reward distribution and learning efficiency in MDPs. This work contributes to a deeper understanding of rewarding process and their practical impact for optimizing agent’s actions in Q-learning.