Scholarlink

<p>Toxicity in large language model (LLM) outputs refers to undesirable attributes in generated text, such as aggression, insults, discrimination, violent tendencies, and gender or racial bias. Researchers have investigated prompt-based explainability methods to identify features in prompts that are likely to elicit toxic outputs, with the goal of informing future efforts to mitigate toxicity. However, as prompt attacks grow more sophisticated, malicious intent is increasingly obfuscated and no longer explicitly manifested at the word level, resulting in the ineffectiveness of traditional word-level explainability approaches. Furthermore, existing semantic-level explainability methods focus primarily on relational analyses of isolated semantic features and lack a quantitative framework that integrates multiple semantic features. As a result, they are unable to accurately identify the key drivers of toxic outputs. To address this challenge, we propose a cascaded attention-based framework for semantic explainability of toxicity in large language models (CASET), which maps toxic outputs to the multi-semantic features of the input prompt. We first construct a semantic representation model that extracts features across four distinct semantic dimensions. Based on the inherent hierarchical structure of natural language, we propose a hierarchy-aligned cascading scheme that organizes attention modules into a unified toxicity prediction model. Finally, we extract the attention weights to assess the relative importance of each semantic feature. We evaluate CASET on the RealToxicityPrompts and PolygloToxicityPrompts datasets using both an open-source LLM (Llama-3-8B) and a closed-source LLM (GPT-4.1-nano). Experimental results show that CASET effectively identifies key semantic features responsible for toxicity and outperforms baseline explainability methods on the <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\varvec{\mathcal {L}}_{\varvec{top}_{\varvec{3}}}\)</EquationSource> </InlineEquation> and <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\varvec{\mathcal {L}}_{\varvec{top}_{\varvec{5}}}\)</EquationSource> </InlineEquation> metrics. Furthermore, case studies and human evaluation validate the practical utility of our framework, demonstrating that its optimization suggestions can significantly reduce toxicity in LLM outputs.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

CASET: a cascaded attention-based framework for semantic explainability of toxicity in large language models

  • 期刊论文

  • 发表日期：

    2026年4月10日

查看全文
  • Chen Chen,
  • Hanyang Xia,
  • Weidong Zhou,
  • Chunhe Xia,
  • Mengyao Liu,
  • Rui Hao,
  • Tianbo Wang

摘要

Toxicity in large language model (LLM) outputs refers to undesirable attributes in generated text, such as aggression, insults, discrimination, violent tendencies, and gender or racial bias. Researchers have investigated prompt-based explainability methods to identify features in prompts that are likely to elicit toxic outputs, with the goal of informing future efforts to mitigate toxicity. However, as prompt attacks grow more sophisticated, malicious intent is increasingly obfuscated and no longer explicitly manifested at the word level, resulting in the ineffectiveness of traditional word-level explainability approaches. Furthermore, existing semantic-level explainability methods focus primarily on relational analyses of isolated semantic features and lack a quantitative framework that integrates multiple semantic features. As a result, they are unable to accurately identify the key drivers of toxic outputs. To address this challenge, we propose a cascaded attention-based framework for semantic explainability of toxicity in large language models (CASET), which maps toxic outputs to the multi-semantic features of the input prompt. We first construct a semantic representation model that extracts features across four distinct semantic dimensions. Based on the inherent hierarchical structure of natural language, we propose a hierarchy-aligned cascading scheme that organizes attention modules into a unified toxicity prediction model. Finally, we extract the attention weights to assess the relative importance of each semantic feature. We evaluate CASET on the RealToxicityPrompts and PolygloToxicityPrompts datasets using both an open-source LLM (Llama-3-8B) and a closed-source LLM (GPT-4.1-nano). Experimental results show that CASET effectively identifies key semantic features responsible for toxicity and outperforms baseline explainability methods on the \(\varvec{\mathcal {L}}_{\varvec{top}_{\varvec{3}}}\) and \(\varvec{\mathcal {L}}_{\varvec{top}_{\varvec{5}}}\) metrics. Furthermore, case studies and human evaluation validate the practical utility of our framework, demonstrating that its optimization suggestions can significantly reduce toxicity in LLM outputs.