Byzantine-Robust Federated Learning via Server-Side Mixtue of Experts

Byzantine-robust Federated Learning focuses on mitigating the impact of malicious clients by developing robust algorithms that ensure reliable model updates while preserving privacy. The key insight of the state-of-the-art approaches entails statistical analysis on the local model updates uploaded by clients, concurrently eliminating any malicious updates prior to their aggregation. Some of these methods also require the assistance of server-side data for a reliable root of trust. However, these methods may not perform well and can even disrupt the normal process when the amount of data on the server-side is limited or the structure of the model is complex. We address this challenge by introducing FLSMoE, a novel Byzantine-robust Federated Learning approach that utilizes a Server-side Mixture of Experts. Our approach introduces a novel methodology by implementing a server-side Mixture of Experts(MoE) model, where the model parameters uploaded by individual clients are considered as expert models. Through the utilization of the gating unit within the MoE, even with low server-side data requirement, we are able to effectively identify and exclude malicious clients by assigning appropriate weights to their contributions. Empirically, we show through an extensive experimental evaluation that FLSMoE with low server-side data requirement can effectively mitigate the threat of malicious clients while also exhibiting greater Byzantine-robustness compared to previous Byzantine-robust Federated Learning approaches.