An Internal-External Constrained Distillation Framework for Continual Semantic Segmentation

Deep neural networks have a notorious catastrophic forgetting problem when training serialized tasks on image semantic segmentation. It refers to the phenomenon of forgetting previously learned knowledge due to the plasticity-stability dilemma and background shift in the segmentation task. Continual Semantic Segmentation (CSS) comes into being to handle this challenge. Previous distillation-based methods only consider the knowledge of the features at the same level but neglect the relationship between different levels. To alleviate this problem, in this paper, we propose a mixed distillation framework called Internal-external Constrained Distillation (ICD), which includes multi-information-based internal feature distillation and attention-based external feature distillation. Specifically, we utilize the statistical information of features to perform internal distillation between the old model and the new model, which effectively avoids interference at the same scale. Furthermore, for the external distillation of features at different scales, we employ multi-scale convolutional attention to capture the relationships among features of different scales and ensure their consistency across old and new tasks. We evaluate our method on standard semantic segmentation datasets, such as Pascal-VOC2012 and ADE20K, and demonstrate significant performance improvements in various scenarios.