Online Continual Learning via Maximal Coding Rate Reduction

Online continual learning aims to acquire new information from a continuous data stream while mitigating catastrophic forgetting of previous knowledge. The fundamental challenge within this learning paradigm lies in the fact that new task-relevant data usually comes in small batches and can only be trained in a single session. To tackle this issue, we propose a new online continual learning solution called OCL-MCR2 (Online Continual Learning via Maximal Coding Rate Reduction), which is based on the principle of maximal coding rate reduction and knowledge distillation. This principle enables the model to learn the most representative and informative essential features in the data of each task by mapping the input data into a lower-dimensional representation space. Consequently, this approach can effectively reduce the feature bias caused by traditional cross-entropy loss, which is recognized as one of the primary factors contributing to catastrophic forgetting. In addition, we introduce knowledge distillation technology, which facilitates the transfer of knowledge from the old model to the new model, further alleviating the catastrophic forgetting problem of the model on previously learned data. Extensive experiments were conducted on three benchmark datasets, and the results show that the method proposed in this paper achieves an average accuracy improvement of 1.2%, 2.9%, and 3.0% on the MNIST, CIFAR-10, and CIFAR-100 datasets, respectively when compared to the online continual learning baselines.