Recent language models continue to improve their performance as the number of parameters is increased. On the other hand, it is known that large language models tend to store whole training samples. It is called memorization. If the training data contains private information, data privacy protection becomes an issue. Deep learning models are also prone to forgetting previously learned data, and large language models have a lower bound on the amount of forgetting. Currently, methods exist to mitigate memorization, such as de-duplication and the method using a learning algorithm that guarantees differential privacy. However, de-duplication does not completely prevent memorization. In this paper, we propose a method that splits a dataset and fine-tunes the pre-trained model in two steps. The method aims to reduce the memorization of the dataset used the first time by taking advantage of the nature of forgetting in large language models. Experimental evaluation results showed that fine-tuning, which updates all parameters, decreases the storage of the first data set by about 2%, although the perplexity decreases by about 0.1, and also decreases the overall storage by about 0.2%. The same trend was not observed for the Adapter with fine-tuning by adding new parameters, and additional verification is considered necessary.