Federated Learning Optimization Algorithm Based on Dynamic Client Scale

Federated learning methods typically learn models from the local iterative updates of a large number of clients. The interest in the impact of client quantity on the training dynamics of federated learning algorithms has been growing in recent years. Increasing the client scale during the training process not only improves data parallelism efficiency but also accelerates the training of federated learning. When optimizing models using a large number of clients, the learning rate needs to adapt to the client scale and the aggregation of updates in order to maximize speed while maintaining model quality. However, the current approach mainly relies on empirically-derived linear learning rate scaling rules, which cannot adapt to the dynamic client scale in federated learning. In this regard, we propose ASNES, an algorithm that dynamically adapts to the client scale in federated learning. By continuously adapting to the client quantity and aggregation of updates, ASNES achieves acceleration for different client scales. In experimental evaluations, ASNES demonstrates favorable performance compared to other benchmark algorithms.