Human-in-the-loop reinforcement learning (HIRL) has proven effective in enhancing the sampling efficiency of deep reinforcement learning by incorporating human knowledge and experience. However, the reliance on expert guidance remains a significant obstacle to its scalability, leading to high human resource costs and limited applicability. This paper introduces an uncertainty-based dynamic weighted experience replay (UDWER) approach, designed to mitigate the dependence on continuous human involvement while maintaining robust learning performance. The proposed method dynamically adjusts the weighting of experience samples based on decision uncertainty, enabling human intervention only when necessary. Experimental results conducted in the Lunar Lander environment demonstrate significant improvements in sample efficiency, reduced human reliance, and faster convergence when compared to existing methods.