Modality-balanced federated learning via cross-client regional prototypes and gradient weighting

Multimodal Federated Learning (MFL) faces significant challenges due to modality imbalance and data heterogeneity, which hinder effective fusion of modality information and introduce modality bias in the global model. Existing client selection techniques often overlook the representational capacity of weaker modalities, and aggregation methods typically fail to consider the varying contributions from different clients. To address these issues, this paper proposes a novel framework combining regional prototypes and gradient-weighted aggregation to enhance modality balance. Specifically, local prototypes are clustered using K-means, and cosine similarity is employed to match each client’s prototype with its closest regional prototype. These prototypes further provide adaptive guidance through a modality enhancement strategy, strengthening the feature representations of weak modalities. A dynamic modality selection mechanism is introduced, where clients’ modality usage is adjusted based on the performance of enhanced modalities and local imbalance rates. Furthermore, a gradient-weighted aggregation method is designed, assigning weights to clients based on the L2 norm of their gradients to mitigate the impact of unbalanced client contributions. Experimental results on the AVE and CREMA-D datasets under varying heterogeneity levels show that the proposed approach improves the accuracy of weaker modalities by up to 1.3–2.3 × , reduces modality bias, and enhances training stability. Compared to traditional methods that use global prototypes and uniform aggregation, the proposed framework maintains more accurate local representations and more robust aggregation, leading to better model performance.