A Survey of Advances in Multimodal Federated Learning with Applications

Data privacy has long been an item of emphasis for personal data. This is especially true for healthcare data, which is often multimodal (i.e., it utilizes in some fashion multiple data streams from multiple sources). In an effort to enhance the knowledge-base of privacy-preserving techniques with respect to multimodal data, we provide a survey of multimodal federated learning (MMFL). Our paper includes a thorough introduction to federated learning as well as a discussion on applications of multimodal federated learning to disease classification, autonomous driving, and human activity recognition, among others. Additionally, we describe various methodological advances in MMFL, a subset of which include extensions to supervised learning, personalization, generative models, data reduction, and feature selection. As a proof-of-concept for MMFL, we also include a novel application of federated learning to a series of physiological signals collected during simulated flights, known as the CogPilot dataset.