Reinforcement Learning from Human Feedback (RLHF) has become a standard approach for aligning large language models (LLMs) with human preferences. However, human preferences are inherently dynamic, often shifting across time and context, which poses significant challenges for reward models in maintaining alignment and stability. In this work, we propose \(\text {RC}^{2}\text {RLHF}\) , a continual RLHF framework that addresses the issue of reward distribution shift in sequential preference learning. Our method consists of two key components. First, a memory-buffer based reward calibration is applied, which projects historical reward data onto the current task distribution, thus mitigating catastrophic forgetting across tasks. Second, a multi-Kullback–Leibler (multi-KL) divergence regularization is applied. It constrains policy updates using two references. One is the initial supervised model, the other is the prior model from the previous task. This constrains policy updates and preserves alignment across tasks. We evaluate our approach on a sequential shifted dataset and demonstrate that \(\text {RC}^{2}\text {RLHF}\) consistently outperforms representative baselines, effectively preserving model performance and alignment under preference shift conditions.