Representation and Granularity Joint Alignment Framework for Multimodal Sarcasm Detection on Social Media

As a sophisticated linguistic phenomenon, sarcasm is prevalent on social media. Multimodal sarcasm detection seeks to identify whether a post is sarcastic by capturing inconsistencies between text and image information. Despite achieving significant success, most existing methods primarily focus on modeling cross-modal relationships through various interaction mechanisms, ignoring the inter-modal gap caused by the representation and granularity of different modalities. To address this limitation, we propose a representation and granularity joint alignment framework. For the representation aspect, we introduce contrastive learning paired with momentum distillation to encode the unimodal information in a common latent embedding space to enhance semantic invariance. For the granularity aspect, we construct a shared multimodal space as a foundational element to reconstruct features of different modalities into a unified granularity. Moreover, bidirectional cross-modal attention is adopted to aggregate global and local features for capturing comprehensive inconsistencies. Extensive experiments on a benchmark dataset demonstrate the superiority of the proposed framework. Our source code is available at https://github.com/yf-lyu/RGAF .