While benchmarks like BBQ have established standards for evaluating social biases in Large Language Models (LLMs), relying solely on translated datasets creates a “safety mirage” for low-resource languages. In this paper, we introduce ViBBQ, a comprehensive benchmark for probing social biases in Vietnamese LLMs, comprising over 61,000 samples constructed through a hybrid pipeline of translation and agentic cultural extension. Our evaluation of five state-of-the-art open-source models reveals a critical “Bias Reversal” phenomenon: while models exhibit aggressive safety over-correction (negative bias) on translated Western contexts, they revert to strong pro-stereotypical prejudices (positive bias) on Vietnam-specific axes generated from local news. These findings confirm that current safety alignment mechanisms are culturally conditioned effective against globalized concepts but permeable to indigenous biases. Among evaluated models, Llama-3.1-8B demonstrates the most robust performance, yet the pervasive inconsistency across contexts underscores the urgent need for culturally grounded safety alignment in Southeast Asian languages.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ViBBQ: Probing Social Biases in Vietnamese LLMs via Translated and Culturally Extended Benchmarks

  • Minh-Phuc Huynh,
  • Phuc-Ha Tu,
  • Hoang Thi Ngoc Trang,
  • Anh-Cuong Le

摘要

While benchmarks like BBQ have established standards for evaluating social biases in Large Language Models (LLMs), relying solely on translated datasets creates a “safety mirage” for low-resource languages. In this paper, we introduce ViBBQ, a comprehensive benchmark for probing social biases in Vietnamese LLMs, comprising over 61,000 samples constructed through a hybrid pipeline of translation and agentic cultural extension. Our evaluation of five state-of-the-art open-source models reveals a critical “Bias Reversal” phenomenon: while models exhibit aggressive safety over-correction (negative bias) on translated Western contexts, they revert to strong pro-stereotypical prejudices (positive bias) on Vietnam-specific axes generated from local news. These findings confirm that current safety alignment mechanisms are culturally conditioned effective against globalized concepts but permeable to indigenous biases. Among evaluated models, Llama-3.1-8B demonstrates the most robust performance, yet the pervasive inconsistency across contexts underscores the urgent need for culturally grounded safety alignment in Southeast Asian languages.