Where to Forget: A New Attention Stability Metric for Continual Learning Evaluation

Continual learning aims to accumulate knowledge from data streams of multiple tasks, which may suffer catastrophic forgetting due to the data incompleteness in each separate training task. To date, the average accuracy and forgetting rate are the two most popular metrics for continual learning evaluation. However, these two metrics only care about the overall increment of mistaken samples when a model updated by the new task is applied to the old tasks, which fails to localize the trajectory of forgetting through multiple training stages. In this paper, we propose a new Attention Stability Metric (ASM) to explicitly illustrate and quantify the forgetting degree of a continual learning model by jointly considering the Changes in Regions of Interest (CRoI) and classification accuracy, which measures the performance of continual learning model in terms of a two-dimensional distance and prefer the models producing consistent changes in terms of RoI and classification accuracy. Experiments show that our evaluation metric offers new insight for analyzing the forgetting characteristics of different continual learning algorithms.