Multimodal Rumor Detection by Using Additive Angular Margin with Class-Aware Attention for Hard Samples

Currently, several factors limit the practicality of multimodal rumor detection (MRD). These include incomplete feature fusion in multimodal data, the weak discriminative power in the softmax-based loss, and the detrimental impact of hard negative samples on the learning process. To address these issues, we propose a MRD framework that combines a supervised contrastive loss with an additive angular margin and incorporates class-aware attention. We propose a multi-layer fusion (MLF) module to enhance the multimodal feature fusion to align and fuse token-level features from text and image modalities. And also, by adding an angular margin to the loss function, we bolster the discriminative power of the contrastive loss. Additionally, the class-aware attention module effectively mitigates the impact of hard negative samples on the supervised contrastive loss. Extensive experiments on three real-world multimodal datasets demonstrate that our proposed learning objective leads to an embedding space that effectively distinguishes between rumors and truths. Furthermore, our work has significantly improved the efficacy of rumor detection, enabling us to promptly identify and curtail rumors’ propagation.