Deep Continual Image Classification Based on Memory Prototype and Energy-Based Feature Synchronizing

Continual learning is aimed to build models that can learn in a human-like manner, in which the models can learn a sequence of tasks coming continuously. The learned models are expected not only to achieve the high efficiency for the current task but also to retain the effectiveness of previous tasks. In this paper, we leverage memory reuse by using prototypes for each class instead of real data. This technique is proved to be effective for continual learning and for addressing both data security concerns and memory buffer issues. Moreover, during the learning process, the hidden feature space of old tasks changes the bias to the hidden feature space of new tasks, leading to the phenomenon of catastrophic forgetting, a major challenge in continual learning. Catastrophic forgetting is the occurrence where a machine learning model’s performance on previous tasks significantly declines as it learns new tasks. In this paper, we propose to use an energy-based model to synchronize the hidden feature space, addressing the issue of catastrophic forgetting. Extensive experimental results demonstrate that our approach achieves higher effectiveness compared to several state-of-the-art methods.