With the rise of social media usage, memes have become a popular way of communication and information sharing. With this hateful memes are also increasing, which are used to spread hate anonymously. Hateful memes combine images and text to convey their offensive messages which pose a unique challenge due to their multimodal nature and reliance on cultural context and humor. While many studies focus on detecting such content, languages with limited resources still face challenges due to a lack of available datasets. This paper introduces a dataset for one of India’s under-resourced languages, specifically Telugu to help address this gap. This dataset consists of 2,131 Telugu memes, consisting of 1,069 hateful memes and 1,062 non-hateful memes. We initially collected 2,500 memes from social media platforms like Facebook, Telegram, and Twitter using web scraping and manual collection. After cleaning the dataset by removing images that contained only text or only visuals without text, we finalized a dataset of 2,131 memes. Annotation and validation were conducted by five native Telugu speakers using a majority vote, followed by a manual review to correct any mislabeling, and a JSON file was created to store metadata for each meme. We evaluate its effectiveness using three types of models text-only, image-only, and multimodal models that analyze both text and images. Our experiments show that the text-only model achieves an accuracy of 54.3%, precision of 53.1%, recall of 55.2%, and an F1 score of 54.1%. The image-only model achieves an accuracy of 60.1%, precision of 58.1%, recall of 60.5%, and an F1 score of 59.6%. Finally, the multimodal model outperforms both with an accuracy of 67.4%, precision of 65.8%, recall of 69.1%, and an F1 score of 67.4%. Our work aims to support further research in this area and improve automated hate speech detection in Telugu.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Benchmark Dataset for Hateful Memes Detection in Telugu

  • Kalyan Reddy,
  • P. Santhi Thilagam

摘要

With the rise of social media usage, memes have become a popular way of communication and information sharing. With this hateful memes are also increasing, which are used to spread hate anonymously. Hateful memes combine images and text to convey their offensive messages which pose a unique challenge due to their multimodal nature and reliance on cultural context and humor. While many studies focus on detecting such content, languages with limited resources still face challenges due to a lack of available datasets. This paper introduces a dataset for one of India’s under-resourced languages, specifically Telugu to help address this gap. This dataset consists of 2,131 Telugu memes, consisting of 1,069 hateful memes and 1,062 non-hateful memes. We initially collected 2,500 memes from social media platforms like Facebook, Telegram, and Twitter using web scraping and manual collection. After cleaning the dataset by removing images that contained only text or only visuals without text, we finalized a dataset of 2,131 memes. Annotation and validation were conducted by five native Telugu speakers using a majority vote, followed by a manual review to correct any mislabeling, and a JSON file was created to store metadata for each meme. We evaluate its effectiveness using three types of models text-only, image-only, and multimodal models that analyze both text and images. Our experiments show that the text-only model achieves an accuracy of 54.3%, precision of 53.1%, recall of 55.2%, and an F1 score of 54.1%. The image-only model achieves an accuracy of 60.1%, precision of 58.1%, recall of 60.5%, and an F1 score of 59.6%. Finally, the multimodal model outperforms both with an accuracy of 67.4%, precision of 65.8%, recall of 69.1%, and an F1 score of 67.4%. Our work aims to support further research in this area and improve automated hate speech detection in Telugu.