Extending Knowledge Distillation for Personalized Federation

Personalized federated learning (PFL) develops a customized model for each local client to mitigate accuracy issues caused by data and system heterogeneity of different local clients. Most works on PFL adopt a centralized federation and therefore suffer from single-point failures or malicious attacks at the global server. In addition, due to system heterogeneity, such as different computing capabilities of local clients, the computational and storage constraints vary significantly among local clients, which limits the centralized model aggregation considerably. To tackle the issues mentioned above, we propose a personalized FL framework, referred to as EKDFL, by extending decoupled knowledge distillation into FL for Personalization. In EKDFL, multiple local clients are in a dynamic connected topology, and local clients are divided into clients with different knowledge bases on data volume. Classical knowledge distillation is decoupled into target class and latent class knowledge distillation. Local clients transfer learnings from knowledge-rich clients to knowledge-poor clients using decoupled knowledge distillation in a peer-to-peer convergent manner. Comprehensive experiments demonstrate that our proposed method outperforms centralized and decentralized FL baselines while significantly mitigating the challenges of heterogeneous data and system.