The rapid development of large-scale language models has garnered widespread interest from both academia and industry. Efficiently applying those models across various domains is now posing a challenge to researchers. High training costs and the relative scarcity of domain-specific data have rendered continual learning on general pretrained language models as one preferable approach. In this paper, we provide a comprehensive analysis and modification of these continual learning strategies for large language models, as they were initially designed for encoder-only architectures. Then a probing algorithm for the token representation shift was proposed to better alleviate forgetting. Additionally, corresponding evaluation metrics were modified for quantitative analysis of our methods. Through the experiment across three different domains, we verified the effectiveness of continual learning and probing algorithms on recent models. Results showed that knowledge distillation outperforms other methods in cross-domain continual learning. Moreover, the introduction of probing can further enhance the accuracy with a relatively small calculation budget.