Although the test-time adaptation (TTA) learning paradigm can against distribution shift, it also results in severe performance degradation on in-distribution (ID) data after adaptation, a phenomenon known as catastrophic forgetting. In this paper, we highlight that the fundamental reason for catastrophic forgetting is that vanilla TTA methods finetune the model without considering the type of input samples. Our key insight is that the model only needs to be finetuned when the input data is from the out-of-distribution (OOD) domain. Motivated by this insight, we proposed a novel and surprisingly effective anti-forgetting test-time adaptation method to address the catastrophic forgetting problem. Specifically, our contributions are two-fold: 1) We propose a novel non-parametric perturbation layer to synthesize OOD data for any input samples, which can enhance model robustness by injecting the synthesized OOD data into the training set and provide supervision signal for OOD detector. 2) We further propose a novel features statistics mean discrepancy metric, which can reliably distinguish the OOD and ID data and finetune model parameters according to the type of input data. Extensive experiments on medical image classification and segmentation tasks demonstrated that our method can address the catastrophic forgetting problem and achieve the SOTA performance.