L2MNet: Enhancing Continual Semantic Segmentation with Mask Matching

Continual semantic segmentation (CSS) aims to continuously learn a semantic segmentation model that incorporates new categories while avoiding forgetting the previously seen categories. However, CSS faces a significant challenge known as weight shift, which leads to the network mistakenly predicting masks belonging to new categories instead of their actual categories. To mitigate this phenomenon, we propose a novel module named mask matching module, which transfers pixel-level prediction task into a mask-level feature matching task by computing the similarity between mask features and prototypes. Further, we introduce a new paradigm and a network called Learn-to-Match (L2M) Net, which alleviates weight shift and gains remarkable improvements on long settings by leveraging mask-level feature matching. Our method can be easily integrated into various network architectures without extra memory and data cost. Experiments conducted on the Pascal-VOC 2012 and ADE20K datasets demonstrate that, particularly on long settings where CSS encounters more challenging settings, our method achieves a remarkable \(10.6\%\) improvement in terms of all mean Intersection over Union (mIoU) and establishes a new state-of-the-art performance in the demanding CSS settings.