Unconstrained Feature Model and Its General Geometric Patterns in Federated Learning: Local Subspace Minority Collapse

Federated Learning is a decentralized approach to machine learning that enables multiple parties to collaborate in building a shared model without centralizing data, but it can face issues related to client drift and the heterogeneity of data. However, there is a noticeable absence of thorough analysis regarding the characteristics of client drift and data heterogeneity in FL within existing studies. In this paper, we reformulate FL using client-class sampling as an unconstrained feature model (UFM), and validates the soundness of UFM in FL through theoretical proofs and experiments. Based on the model, we explored the potential information loss, the source of client drifting, and general geometric patterns in FL, called local subspace minority collapse. Through theoretical deduction and experimental verification, we provide support for the soundness of UFM and observe its predicted phenomenon, neural collapse.