Correlation Aware Imbalanced Multimodal Fusion in IoT Environment
摘要
With the rapid advancement of the Internet of Things (IoT) and related technologies, contemporary systems are increasingly required to process diverse multimodal data from external sources. Faced with vast amounts of multimodal content and semantic heterogeneity, these systems necessitate efficient multimodal fusion mechanisms to enhance their intelligent perception capabilities. Importantly, modalities are not independent; their inter-modal correlations significantly influence service coordination efficiency. Besides, dynamic environmental variations introduce noise and quality fluctuations across modalities. This modality imbalance presents substantial challenges to system robustness: such inter-modal instability disrupts correlation consistency, thereby undermining the reliability of service coordination and hindering consistent perceptual performance in complex environments. To achieve unified multimodal fusion, effective service coordination, and adaptive modality integration for constructing intelligent, efficient, and reliable IoT systems, this paper proposes a Correlation aware Multimodal Fusion Network named CMMFussionNet. Our network explicitly addresses semantic discrepancies among multimodal data while focusing on mitigating modality imbalance to enable holistic system synergy. Experiments conducted on large-scale real-world open datasets demonstrate that CMMFussionNet outperforms existing benchmarks by 2.6%. This finding verifies that our approach not only enhances intra-model feature representation but also effectively resolves issues related to modality imbalance.