Quantized SGD in Federated Learning: Communication, Optimization and Generalization

Federated Learning (FL) has gained significant attention due to its impressive scalability properties and its ability to preserve data privacy. To improve the communication efficiency of FL, quantization techniques have emerged as commonly used approaches. However, the introduction of randomized quantization in FL can introduce additional variance, impacting the accuracy of the models. Furthermore, few studies in the existing literature have explored the impact of quantization on the generalization ability of FL algorithms. In this paper, we focus on examining the interplay among key factors in the widely used distributed Stochastic Gradient Descent (SGD) algorithm with quantization. Specifically, we investigate the relationship between quantization level, optimization error, and generalization performance. For convex objectives, our main results reveal several trade-offs between communication efficiency, optimization error, and generalization performance. In the case of non-convex objectives, our theoretical findings indicate that the quantization level has a more significant impact on the generalization ability compared to convex cases. Moreover, our derived generalization bounds for non-convex objectives suggest that early stopping may be necessary to ensure a certain level of generalization accuracy, even when the step size in SGD is very small. Finally, we conduct several numerical experiments utilizing logistic models and deep neural networks to validate and support our theoretical findings.