Decoupled Knowledge Distillation in Data-Free Federated Learning

As the digital age advances, enormous mobile devices are interconnected with the Internet, resulting in a huge amount of distributed data. In response to this demand, federated learning (FL) arises. FL permits clients to collaborate on training a network model under the coordination of a central server by merely transmitting model parameters instead of private data with the central server, maintaining the decentralization and dispersion of the training data. However, distributed model training brings out the challenge of system heterogeneity and data heterogeneity. To mitigate these issues, numerous studies have employed methods such as regularized local loss, Bayesian approaches, meta-learning, and multi-task learning, which inevitably incur higher overhead. Consequently, achieving a trade-off between model accuracy and communication overhead has become a focus of FL algorithm research. This paper proposes a new method, Federated Learning via Decoupled Knowledge Distillation and Generative Model(FedDKDGen). We conduct massive experiments to explore the impact of statistical heterogeneity, communication rounds, and active-user ratio on model performance, respectively. Experimental results demonstrate that FedDKDGen performs admirably on EMNIST and MNIST, better exploiting the high-level semantics of the logits layer. As a result, the model accuracy advantage becomes increasingly pronounced as the data heterogeneity grows. Moreover, the model accuracy steadily outperforms the FL algorithm compared to, accounting for the lower communication cost due to the faster convergence rate. This allows FedDKDGen to achieve the same model accuracy as other algorithms with fewer communication rounds.