Small language models (SLMs) are gaining traction as attractive alternatives to large-scale LLMs due to their low computational cost, suitability for on-device inference, and reduced environmental footprint. As efforts intensify to compress and distill powerful language capabilities into smaller architectures, SLMs are increasingly deployed in edge devices, embedded systems, and offline applications. However, the safety and alignment properties of these models remain underexplored. In this work, we identify and characterise a critical blind spot: the misalignment of SLMs and their vulnerability to unsafe behaviour and jailbreak attacks. We introduce a two-phase evaluation pipeline: first, we test a suite of direct malicious prompts against a range of SLMs; secondly, we apply more sophisticated, tailored jailbreak attacks typically used to circumvent the safety filters of larger, more aligned models. Notably, some SLMs that initially appear safe under naive testing, turn out to be highly susceptible to jailbreak prompts—revealing a discrepancy between perceived and actual safety. To address this, we propose a lightweight mitigation strategy tailored for resource-constrained models, capable of intercepting jailbreak attempts at inference time without requiring retraining. Our findings underscore the urgent need to extend safety auditing, evaluation benchmarks, and mitigation strategies to the rapidly growing class of small-scale language models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Small but Dangerous: Evaluating and Mitigating Jailbreak Vulnerabilities in Small Language Models

  • Leonardo Piano,
  • Claudia Battistin,
  • Jeriek Van den Abeele,
  • Livio Pompianu

摘要

Small language models (SLMs) are gaining traction as attractive alternatives to large-scale LLMs due to their low computational cost, suitability for on-device inference, and reduced environmental footprint. As efforts intensify to compress and distill powerful language capabilities into smaller architectures, SLMs are increasingly deployed in edge devices, embedded systems, and offline applications. However, the safety and alignment properties of these models remain underexplored. In this work, we identify and characterise a critical blind spot: the misalignment of SLMs and their vulnerability to unsafe behaviour and jailbreak attacks. We introduce a two-phase evaluation pipeline: first, we test a suite of direct malicious prompts against a range of SLMs; secondly, we apply more sophisticated, tailored jailbreak attacks typically used to circumvent the safety filters of larger, more aligned models. Notably, some SLMs that initially appear safe under naive testing, turn out to be highly susceptible to jailbreak prompts—revealing a discrepancy between perceived and actual safety. To address this, we propose a lightweight mitigation strategy tailored for resource-constrained models, capable of intercepting jailbreak attempts at inference time without requiring retraining. Our findings underscore the urgent need to extend safety auditing, evaluation benchmarks, and mitigation strategies to the rapidly growing class of small-scale language models.