Node selection for model quality optimization in hierarchical federated learning based on deep reinforcement learning

In Hierarchical Federated Learning (HFL), data sample sizes and distribution of different clients vary greatly. Due to the heterogeneity of the data, it is crucial to select appropriate clients to participate in model training while ensuring the model quality of HFL. We investigate the problem of optimizing client selection for model quality. We investigate the impact of Non-Independent and Identically Distributed data on HFL and found that selecting clients based on losses can improve model quality. Thus, We propose a client selection method based on Client Quality Records (CS-Loss), utilizing client losses. Since selecting clients to participate in model training at each iteration round results in changes to client losses and model parameters, the process becomes dynamic. Therefore, we formulate the client selection problem as a Markov Decision Process and design an algorithm based on Synchronous Advantage Actor-Critic (CS-A2C) to address it. Simulation results demonstrate that the CS-A2C algorithm outperforms both the existing FedAvg algorithm and Favor algorithm on the MNIST dataset. On the CIFAR-10 dataset, the proposed CS-A2C algorithm can improve model accuracy by 13% and 7% respectively.