Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, but conventional parameter aggregation often leads to unstable convergence and poor generalization under non-IID data. Federated Knowledge Distillation (FKD) alleviates communication overhead by transferring soft predictions instead of parameters, yet most existing methods use uniform aggregation and ignore the varying reliability of client knowledge. To address this issue, we propose WMT-FedKD, a Weighted Multi-Teacher Federated Knowledge Distillation framework that enhances robustness and communication efficiency through a reliability-aware weighting mechanism and an adaptive multi-teacher fusion strategy. Each client is assigned a dynamic reliability weight based on prediction similarity, confidence, and temporal stability, ensuring that more reliable knowledge contributes more effectively to the global model. In addition, an Exponential Moving Average (EMA) teacher maintains long-term stability, while a lightweight gating network adaptively balances the influences of client and EMA teachers. Extensive experiments on MNIST, EMNIST, SVHN, and FashionMNIST show that WMT-FedKD achieves up to 13.2% higher accuracy and reduces communication rounds by 40–70% compared with existing FL and FKD baselines. These results demonstrate that WMT-FedKD effectively stabilizes training, accelerates convergence, and improves generalization under heterogeneous federated environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

The Power of Weighting: Multi-teacher Distillation for Communication-Efficient Federated Learning

  • Ruojia Zhang,
  • Weijia Feng,
  • Tongtong Su,
  • Wentao Wang,
  • Fengtao Sun,
  • Chenyang Wang,
  • Chongke Bi

摘要

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, but conventional parameter aggregation often leads to unstable convergence and poor generalization under non-IID data. Federated Knowledge Distillation (FKD) alleviates communication overhead by transferring soft predictions instead of parameters, yet most existing methods use uniform aggregation and ignore the varying reliability of client knowledge. To address this issue, we propose WMT-FedKD, a Weighted Multi-Teacher Federated Knowledge Distillation framework that enhances robustness and communication efficiency through a reliability-aware weighting mechanism and an adaptive multi-teacher fusion strategy. Each client is assigned a dynamic reliability weight based on prediction similarity, confidence, and temporal stability, ensuring that more reliable knowledge contributes more effectively to the global model. In addition, an Exponential Moving Average (EMA) teacher maintains long-term stability, while a lightweight gating network adaptively balances the influences of client and EMA teachers. Extensive experiments on MNIST, EMNIST, SVHN, and FashionMNIST show that WMT-FedKD achieves up to 13.2% higher accuracy and reduces communication rounds by 40–70% compared with existing FL and FKD baselines. These results demonstrate that WMT-FedKD effectively stabilizes training, accelerates convergence, and improves generalization under heterogeneous federated environments.