The Power of Weighting: Multi-teacher Distillation for Communication-Efficient Federated Learning
摘要
Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, but conventional parameter aggregation often leads to unstable convergence and poor generalization under non-IID data. Federated Knowledge Distillation (FKD) alleviates communication overhead by transferring soft predictions instead of parameters, yet most existing methods use uniform aggregation and ignore the varying reliability of client knowledge. To address this issue, we propose WMT-FedKD, a Weighted Multi-Teacher Federated Knowledge Distillation framework that enhances robustness and communication efficiency through a reliability-aware weighting mechanism and an adaptive multi-teacher fusion strategy. Each client is assigned a dynamic reliability weight based on prediction similarity, confidence, and temporal stability, ensuring that more reliable knowledge contributes more effectively to the global model. In addition, an Exponential Moving Average (EMA) teacher maintains long-term stability, while a lightweight gating network adaptively balances the influences of client and EMA teachers. Extensive experiments on MNIST, EMNIST, SVHN, and FashionMNIST show that WMT-FedKD achieves up to 13.2% higher accuracy and reduces communication rounds by 40–70% compared with existing FL and FKD baselines. These results demonstrate that WMT-FedKD effectively stabilizes training, accelerates convergence, and improves generalization under heterogeneous federated environments.