Entrofuse: Clustered Federated Learning Through Entropy Approach

Conventional machine learning method typically relies on collecting vast quantities of data, which usually results in serious private information leakage and a huge communication burden. To tackle this severe challenge, Federated Learning (FL), which served as a novel paradigm of distributed machine learning, is recently proposed. Under the framework of FL, clients cooperatively train a shared global model with their own data and transmit the model parameter to the central server while keeping their private data localized. However, FL still encounters some limitations, particularly in confronting non-independent and non-identically distributed (Non-IID) data which results in poor model performance. In light of the above concerns, we propose an entropy-based clustering federated learning model named Entrofuse, which aims to partition the clients into different clusters characterized by data distribution and subsequently, the model training process performs within each cluster. As it is hard to acquire the distribution of data samples, we adopt Kernel Density Estimation (KDE) method to estimate the data distribution of heterogeneous clients. Our approach takes into account both entropy and vector angle of the model parameter, and proves the rationality of our method through rigorous theoretical analysis. Experimental results show that our proposed method is superior to the non-clustered case on the EMNIST dataset and significantly improves the accuracy by 10% to 12%.