Multimodal Metadata Augmentation for Federated Learning in Medical Applications

Multimodal Metadata Augmentation (MMA) is explored in the context of federated learning (FL) for image classification tasks in medical applications. The impact of augmenting medical image data with additional metadata, such as partial text opinions, in multimodal configuration, on the performance of deep neural network (DNN) models. The medical PathMNIST and general-purpose CIFAR10 datasets are used for FL experimentation on the highly imbalanced datasets. The results demonstrate that MMA can significantly improve classification accuracy for PathMNIST by various ensembling methods including majority voting, averaging, maximizing, and their versions without taking into account potential outliers. The results obtained suggest that DNN with MMA has potential applications in secure and privacy-preserving FL for medical image analysis.