Scholarlink

<p>Large Language Models (LLMs) struggle to keep up with the fast-changing nature of real-world information, as their pre-trained knowledge quickly becomes outdated. This work addresses the challenge of keeping LLMs up to date with factual knowledge (adaptation) while avoiding forgetting the relevant existing knowledge. Leveraging temporally-aligned Wikipedia and Wikidata dumps, we extract a continuous data stream and evaluate the performance of an incrementally trained GPT-2 across different time periods. Additionally, we extend our analysis to real-world news data using the RealTimeData dataset, examining how LLMs respond to novel facts, such as the COVID-19 pandemic. Our methodology includes synthetic data generation and <span>SmartReview</span>, a continual learning strategy that avoids forgetting by rehearsing on a carefully selected subset of the old data. Experimental results highlight that pretrained models require continual learning and demonstrate the effectiveness of replay-based approaches in mitigating forgetting. In particular, <span>SmartReview</span> provides a strong replay-based baseline that limits forgetting and enhances adaptation. This work advances the study of continual learning in LLMs, offering insights into the development of more temporally-aware and reliable AI systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adapting language models with continual learning for temporal drifts

  • 期刊论文

  • 发表日期：

    2026年3月28日

查看全文
  • Antonio Carta,
  • Alberto Roberto Marinelli,
  • Lucia C. Passaro

摘要

Large Language Models (LLMs) struggle to keep up with the fast-changing nature of real-world information, as their pre-trained knowledge quickly becomes outdated. This work addresses the challenge of keeping LLMs up to date with factual knowledge (adaptation) while avoiding forgetting the relevant existing knowledge. Leveraging temporally-aligned Wikipedia and Wikidata dumps, we extract a continuous data stream and evaluate the performance of an incrementally trained GPT-2 across different time periods. Additionally, we extend our analysis to real-world news data using the RealTimeData dataset, examining how LLMs respond to novel facts, such as the COVID-19 pandemic. Our methodology includes synthetic data generation and SmartReview, a continual learning strategy that avoids forgetting by rehearsing on a carefully selected subset of the old data. Experimental results highlight that pretrained models require continual learning and demonstrate the effectiveness of replay-based approaches in mitigating forgetting. In particular, SmartReview provides a strong replay-based baseline that limits forgetting and enhances adaptation. This work advances the study of continual learning in LLMs, offering insights into the development of more temporally-aware and reliable AI systems.