Continual learning faces the critical challenge of catastrophic forgetting when models learn new knowledge. In this paper, we introduce a method that is based on mechanics (CLM), an approach that applies conservation principles from mechanics to balance the learning dynamics between new and previously acquired information. By establishing analogies between model parameters and physical quantities (mapping Fisher Information to mass and parameter gradients to velocity), our method enables more effective parameter integration across sequential learning tasks. We conducted experiments using the ViT-B/16 architecture on CIFAR-100 and ImageNet-R datasets, where CLM demonstrates comparative performance with some existing related techniques. Our approach achieves a 0.31% improvement in average accuracy on CIFAR100 and a substantial 2.04% gain on ImageNet-R over state-of-the-art methods. These results demonstrate that our proposed method is effective in preserving previous knowledge while efficiently accommodating new information.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CLM: Momentum and Torque Conservation for Robust Continual Learning

  • Thi Quynh-Trang Pham,
  • Van-Toan Phan,
  • Xuan-Hung Ho,
  • Tri-Thanh Nguyen

摘要

Continual learning faces the critical challenge of catastrophic forgetting when models learn new knowledge. In this paper, we introduce a method that is based on mechanics (CLM), an approach that applies conservation principles from mechanics to balance the learning dynamics between new and previously acquired information. By establishing analogies between model parameters and physical quantities (mapping Fisher Information to mass and parameter gradients to velocity), our method enables more effective parameter integration across sequential learning tasks. We conducted experiments using the ViT-B/16 architecture on CIFAR-100 and ImageNet-R datasets, where CLM demonstrates comparative performance with some existing related techniques. Our approach achieves a 0.31% improvement in average accuracy on CIFAR100 and a substantial 2.04% gain on ImageNet-R over state-of-the-art methods. These results demonstrate that our proposed method is effective in preserving previous knowledge while efficiently accommodating new information.