Scholarlink

<p>Hate speech on social media poses a serious threat to online safety and social harmony, especially in multilingual and low-resource contexts such as Indonesia. The challenge is further amplified by the increasing prevalence of multimodal contents that combine both text and images, which are often used to convey harmful messages more subtly. However, most existing research focuses only on text-based detection, leaving a gap in understanding how visual information contributes to hate speech utterance. This study aims to comprehensively investigate hate speech detection on Indonesian social media within a multimodal setting. We constructed a novel multimodal hate speech dataset that includes tweet text and accompanying images collected from X, with annotations provided by experts. For evaluation, we explore five hate speech detection approaches: unimodal baselines, early fusion, late fusion, multimodal large language models (MLLMs), and textualization-based fusion. Additionally, we also employed traditional machine learning and transformer-based models for more comprehensive assessment. Our experiment results show that textual information alone remains highly predictive, with a strong baseline F1-score of 0.803 achieved by <Emphasis FontCategory="NonProportional">XLM-RoBERTa</Emphasis>. MLLMs such as <Emphasis FontCategory="NonProportional">Gemini 2.5 Flash</Emphasis> and <Emphasis FontCategory="NonProportional">GPT−5.2</Emphasis> demonstrate competitive zero-shot performance (up to F1 = 0.799), though with variability across languages and prompting strategies. Our experiment also uncovered that combining raw image and text via early or late fusion yields only marginal gains. In contrast, our proposed textualization approach by transforming images into descriptive text using <Emphasis FontCategory="NonProportional">GPT-4o</Emphasis> and concatenating them with tweet text achieves the best overall result with an F1-score of 0.833. These findings suggest that structured textual representations of visual content can significantly enhance multimodal classification performance while improving interpretability. This work provides valuable insights for future research on hate speech detection, especially in resource-scarce and linguistically diverse regions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Reading between modalities: multimodal hate speech detection in low-resource Indonesian social media

  • 期刊论文

  • 发表日期：

    2026年4月7日

查看全文
  • Endang Wahyu Pamungkas,
  • Arida Ferti Syafiandini,
  • Dian Purworini,
  • Widi Widayat,
  • Divi Galih Prasetyo Putri,
  • Ikhlasul Amal,
  • Min Song

摘要

Hate speech on social media poses a serious threat to online safety and social harmony, especially in multilingual and low-resource contexts such as Indonesia. The challenge is further amplified by the increasing prevalence of multimodal contents that combine both text and images, which are often used to convey harmful messages more subtly. However, most existing research focuses only on text-based detection, leaving a gap in understanding how visual information contributes to hate speech utterance. This study aims to comprehensively investigate hate speech detection on Indonesian social media within a multimodal setting. We constructed a novel multimodal hate speech dataset that includes tweet text and accompanying images collected from X, with annotations provided by experts. For evaluation, we explore five hate speech detection approaches: unimodal baselines, early fusion, late fusion, multimodal large language models (MLLMs), and textualization-based fusion. Additionally, we also employed traditional machine learning and transformer-based models for more comprehensive assessment. Our experiment results show that textual information alone remains highly predictive, with a strong baseline F1-score of 0.803 achieved by XLM-RoBERTa. MLLMs such as Gemini 2.5 Flash and GPT−5.2 demonstrate competitive zero-shot performance (up to F1 = 0.799), though with variability across languages and prompting strategies. Our experiment also uncovered that combining raw image and text via early or late fusion yields only marginal gains. In contrast, our proposed textualization approach by transforming images into descriptive text using GPT-4o and concatenating them with tweet text achieves the best overall result with an F1-score of 0.833. These findings suggest that structured textual representations of visual content can significantly enhance multimodal classification performance while improving interpretability. This work provides valuable insights for future research on hate speech detection, especially in resource-scarce and linguistically diverse regions.