Fisher with Class-Contribution in Non-iid Federated Continual Learning

Federated continual learning, that is, the client’s task is to learn a series of incremental data. However, the classic method Fisher matrix reduces the representativeness when facing non-independent and identically distributed data, which limits the capture of task importance. First, we improve the calculation method of the Fisher matrix by considering the contribution of each class, thereby improving the ability to capture task importance. Secondly, we raise a Fisher’s integration strategy to redistribute the Fisher matrix weight, according to the differential contribution of each client Fisher and the old task Fisher to maintain the local model’s representation of the current task. Experimental results show that our proposed algorithm exhibits superior performance on the federated classic dataset. Extensive experiments on fine-grained classification are conducted demonstrating the effectiveness of our approach. Particularly, CNLNet achieves competitive results with the most recent state-of-the-art on three benchmark datasets.