Retrieval-Based Multimodal Data Augmentation for Multimodal Information Extraction in Social Media

Recently, multimodal information extraction (MIE) has attracted increasing attention in social media understanding. The data augmentation methods can effectively address the unique challenges of information extraction on social media, such as data sparsity and insufficient semantics. However, existing data-augmented methods have two weaknesses: (1) existing methods are based on predefined rules or generative models, resulting in the generation of synthetic data that has limited diversity and differs from real-world data; (2) current approaches predominantly focus on text augmentation, overlooking the potential benefits of augmenting image data. To address these issues, we propose a retrieval-based multimodal data augmentation (RMDA) approach by leveraging the social media domain’s massive data volumes and high retrievability, which obtains real-world multimodal posts related to the original data as augmented examples through retrieval. We have conducted extensive experiments to demonstrate the effectiveness of our method and demonstrate that it offers significant advantages in both efficiency and performance compared to augmentation methods based on large language models.