<p>This paper introduces a new reinforcement learning framework that integrates three key methods to optimize action selection into the Temporal Difference Model Predictive Control (TD-MPC) (as reported by Hansen In: international conference on machine learning, 2022), a leading model-based reinforcement learning algorithm, and improve its learning performance in complex environments. First, dual-policy action selection combines the strengths of both model-based and model-free reinforcement learning, enabling more effective and flexible decision-making. Second, policy annealing is introduced to dynamically balance the contributions of these dual policies, preventing excessive exploitation and ensuring robust learning across varying conditions. Third, curiosity-driven exploration is employed to strike an optimal balance between exploration and exploitation, encouraging the agent to explore less familiar states and enhancing overall learning effectiveness. Ultimately, we propose a new scheme, called TD-MPC++, which incorporates the above three key methods into TD-MPC, a leading model-based reinforcement learning algorithm. Through extensive empirical evaluation in the varied environments of the DeepMind Control Suite, we demonstrate that TD-MPC++ achieves not only marked improvements in both sample efficiency and overall training performance when compared to its predecessor, TD-MPC, but also outperforms other state-of-the-art algorithms, including Dreamer V3 and Soft Actor-Critic. Notably, TD-MPC++ achieves superior performance compared to TD-MPC without introducing any additional learning tasks to the models or increasing time complexity.</p>