Multimodal Fake News Detection Through Semantically Enriched Image Captions
摘要
Traditional fake news detection (FND) approaches are mainly based on textual data that often fails to capture the nuanced nature of misinformation. Recent multimodal methods improve the performance of FND by integrating text and images, but the computational complexity is high. To overcome this issue, this paper proposes a lightweight model that enriches textual information through entity-based image caption enhancement. Specifically, captions of images are generated using EfficientNet-V2 and CLIP; after that, visual entities are incorporated to enrich the textual representation. This enhanced caption is used for feature extraction using the BERT; it eliminates the need for raw image data while still preserving multimodal context. In addition to this, we compute three image–text similarity (ITS), as textual similarity (TS), semantic similarity (SS), and contextual similarity (CS) to analyze the alignment between text and image. It is observed that fake news often has a higher similarity score than real news. Further, the enhanced captions and similarity score are considered for classification. Experimental evaluation on Weibo, Gossipcop, and Fakeddit datasets shows state-of-the-art performance, achieving Matthews Correlation Coefficient (MCC) scores of 0.915, 0.942, and 0.900, respectively. Overall, the proposed model demonstrates that incorporating visual cues into enriched textual features provides an effective and computationally efficient solution for FND.