Federated Incremental Learning Method Based on Memory Replay and Self-distillation Technology
摘要
Federated learning (FL), as a distributed machine learning paradigm, can realize multi-party collaborative training models while protecting user privacy and security. However, traditional federated learning methods usually assume that the data distribution of participants is static, ignoring the characteristics of dynamic expansion of data scale and increasing categories in actual scenarios. This limitation causes the model to face the problem of catastrophic forgetting in the process of incremental learning, that is, the introduction of new knowledge will overwrite the memory of old knowledge. To address this challenge, this paper proposes a novel "memory-enhanced adaptive distillation" (MEAD) method. MEAD uses the scores of new categories predicted by the current model to enrich the scores of old categories of the historical model, and adjusts the scale difference so that the sum of the scores of new and old categories is 1, constructing composite knowledge for self-distillation. In addition, this method also adds a memory replay module and uses a generative adversarial network (GAN) to generate samples of old categories.By using memory replay technology to retain information of old categories, it also uses generated samples to indirectly review historical tasks to reduce catastrophic forgetting. Experimental results show that MEAD reduces the average forgetting rate by 6.49%–24.27% compared to mainstream methods on multiple benchmark datasets, and improves the global accuracy by 0.37%–20.37%.