This paper presents DialRet, a dialogue-specific language model that effectively retains previous multi-session dialogues and provides detailed responses. Unlike previous works that rely on memory modules, we leverage the sufficiently long context length of recent language models and instead focus on instruction-tuning based on our proposed eight tasks, including dialogue generation, summarization, speaker relation extraction, time interval estimation, etc. These tasks help a model to have a better understanding of the previous multi-session conversations. Furthermore, we present MSC-Bench, a benchmark specifically designed to evaluate the response quality of multi-session dialogue systems in terms of four key criteria: memorability, specificity, engagingness, and humanness; they assess dialogue models in terms of how well they recall detailed information from previous multi-session dialogues and how specifically they can respond, while faithfully achieving core goals of conversations. The experimental results indicate that DialRet outperforms existing dialogue models on both MSC-Bench and traditional evaluation metrics, demonstrating superior dialogue retention and understanding as well as higher response quality in multi-session scenarios. More details of our project are available at: https://huggingface.co/DILAB-HYU/DialRet .