Research on Multi-modal Class Incremental Learning Methods

In the scenario of incremental learning, the catastrophic forgetting issue is widely prevalent, as the forgetting of old knowledge significantly impacts the model’s performance across all tasks. To tackle the issue of forgetting old knowledge due to prototype shift, this paper proposes a class incremental learning method based on CLIP multi-modal fusion–CLTCIL. This method introduces semantic information in the feature extraction stage of the original model, enhancing the model’s representation of images. Specifically, a parallel CLIP semantic extraction network is integrated into the feature extraction network of the original model, thereby improving the model’s ability to represent images and alleviating the forgetting of old classes. Experiments conducted on the CIFAR100 and ImageNet100 datasets demonstrate that the introduced CLTCIL class incremental learning method effectively reduces catastrophic forgetting of prior knowledge and enhances the model’s classification accuracy in incremental learning scenarios.