Large Language Models (LLMs) are constrained by their lack of a long-term memory mechanism, which hinders their ability to maintain context over extended periods and leads to the loss of crucial historical information. To address this limitation, in this paper, we propose the Self-Controlled Memory (SCM) framework to enhance the ability of LLMs to maintain long-term memory and recall relevant information. Our SCM framework comprises three key components: an LLM-based agent serving as the backbone of the framework, a memory stream storing agent memories, and a memory controller updating memories and determining when and how to use the memories from the memory stream. Furthermore, we annotate a dataset, MemoEval, to assess the efficiency of SCM in utilizing memories and processing lengthy inputs. The MemoEval dataset covers three tasks: long-term dialogues, book summarization, and meeting summarization. Experimental results reveal that our SCM framework significantly increases overall accuracy by about 40% compared to vanilla ChatGPT on the long-term dialogue task (code: https://github.com/wbbeyourself/SCM4LLMs ).