Scholarlink

<p>This paper presents a systematic evaluation of Large Language Models’ (LLMs) behavior on encrypted texts to discuss its safety implications. We introduce a two-dimensional evaluation framework that separately assesses <i>early instruction refusal</i> (whether models refuse harmful-looking instructions) and <i>generation safety</i> (whether models suppress harmful content generation). Previous works have demonstrated that models possessing decryption capabilities are susceptible to <i>under-generalization attacks</i>, where safety mechanisms trained on natural language fail to generalize to encrypted formats. In this work, we show that such scenarios inevitably result in failure along at least one safety dimension: either the generation of unsafe responses (inadequate generation safety) or the over-refusal of legitimate requests (excessive early refusal). Based on these findings, we evaluate a number of pre-LLM and post-LLM safeguards in the encryption schemes where models possess decryption capability (i.e., easy ciphers) and our findings reveal that: (1) most models struggle at balancing both dimensions effectively—prioritizing either instruction refusal or response suppression; (2) pre-LLM defenses fail due to lack of semantic comprehension of encrypted content; (3) post-LLM defenses achieve strong harmful response suppression but are susceptible to over-refusal. This work contributes systematic evaluation methodology and identifies fundamental trade-offs in current safety approaches by analyzing encrypted content.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Should LLM Safety be More Than Refusing Harmful Instructions?

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月22日

查看全文
  • Utsav Maskey,
  • Mark Dras,
  • Usman Naseem

摘要

This paper presents a systematic evaluation of Large Language Models’ (LLMs) behavior on encrypted texts to discuss its safety implications. We introduce a two-dimensional evaluation framework that separately assesses early instruction refusal (whether models refuse harmful-looking instructions) and generation safety (whether models suppress harmful content generation). Previous works have demonstrated that models possessing decryption capabilities are susceptible to under-generalization attacks, where safety mechanisms trained on natural language fail to generalize to encrypted formats. In this work, we show that such scenarios inevitably result in failure along at least one safety dimension: either the generation of unsafe responses (inadequate generation safety) or the over-refusal of legitimate requests (excessive early refusal). Based on these findings, we evaluate a number of pre-LLM and post-LLM safeguards in the encryption schemes where models possess decryption capability (i.e., easy ciphers) and our findings reveal that: (1) most models struggle at balancing both dimensions effectively—prioritizing either instruction refusal or response suppression; (2) pre-LLM defenses fail due to lack of semantic comprehension of encrypted content; (3) post-LLM defenses achieve strong harmful response suppression but are susceptible to over-refusal. This work contributes systematic evaluation methodology and identifies fundamental trade-offs in current safety approaches by analyzing encrypted content.