Multimodal Fusion with External Knowledge (MFEK)
摘要
Information fragmentation is a defining property of social media: early posts are short, informal, and often omit crucial actors, locations, or explicit event keywords. In this chapter, we present Multimodal Fusion with External Knowledge (MFEK), a framework that treats missing context as a first-class bottleneck and recovers it by integrating external knowledge with multimodal evidence. We first analyze why fragmentation causes predictable failures such as missed early signals, wrong grounding, and shallow clustering. We then introduce the MFEK architecture as a dual-source knowledge integration pipeline, including multimodal information extraction, knowledge retrieval from complementary sources, and knowledge-aware fusion via attention-based contextualization. Next, we present the Social Event Detection (SED) dataset, describing its collection, annotation, and key statistics designed to reflect realistic context scarcity. Finally, we report extensive experiments, including strong baselines and ablation analyses, to quantify when knowledge improves detection robustness and clustering stability and when noisy knowledge can hurt, offering practical guidance for deployment under real-time constraints.