The proliferation of large language models (LLMs) in intelligent applications has introduced critical safety challenges concerning input and output content moderation. Current guardrail solutions suffer from inadequate performance in small-parameter models while excessive resource overhead in large-parameter models, limited capacity for fine-grained rule customization, and improvable out-of-distribution generalization. To overcome these limitations, we introduce SCoT Guard, a novel reasoning-based guardrail framework that dynamically acquires fine-grained judgment criteria through contextual learning from examples. Our approach leverages a meticulously curated training dataset of 40,000 high-quality instances generated using DeepSeek-R1, incorporating comprehensive safety detection reasoning chains and rule-learning mechanisms. We implement LoRA fine-tuning on the Qwen2.5-1.5B-Instruct model to achieve optimal performance. At inference time, SCoT Guard utilizes Retrieval-Augmented Generation (RAG) to identify similar examples, extracts their underlying judgment principles, and applies these learned rules to evaluate the new sample, thereby accommodating evolving safety requirements. Comprehensive experimental evaluation demonstrates that SCoT Guard substantially surpasses 9 baseline methods across 6 benchmarks (e.g., OpenAI Moderation, ToxicChat, BeaverTails). These results confirm the effectiveness of integrating chain-of-thought reasoning with RAG in enhancing the performance, interpretability, and adaptability of guardrail models. With its compact 1.5B parameter architecture, SCoT Guard enables direct deployment on resource-limited IoT edge devices, facilitating real-time local inference while eliminating cloud dependency and ensuring data privacy. Both the training dataset and model weights will be open-sourced.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SCoT Guard: A Safety Chain of Thought Guardrail Model

  • Yuxuan Lin,
  • Shi Liu,
  • Dongqin Liu,
  • Wei Mi,
  • Xuehai Tang

摘要

The proliferation of large language models (LLMs) in intelligent applications has introduced critical safety challenges concerning input and output content moderation. Current guardrail solutions suffer from inadequate performance in small-parameter models while excessive resource overhead in large-parameter models, limited capacity for fine-grained rule customization, and improvable out-of-distribution generalization. To overcome these limitations, we introduce SCoT Guard, a novel reasoning-based guardrail framework that dynamically acquires fine-grained judgment criteria through contextual learning from examples. Our approach leverages a meticulously curated training dataset of 40,000 high-quality instances generated using DeepSeek-R1, incorporating comprehensive safety detection reasoning chains and rule-learning mechanisms. We implement LoRA fine-tuning on the Qwen2.5-1.5B-Instruct model to achieve optimal performance. At inference time, SCoT Guard utilizes Retrieval-Augmented Generation (RAG) to identify similar examples, extracts their underlying judgment principles, and applies these learned rules to evaluate the new sample, thereby accommodating evolving safety requirements. Comprehensive experimental evaluation demonstrates that SCoT Guard substantially surpasses 9 baseline methods across 6 benchmarks (e.g., OpenAI Moderation, ToxicChat, BeaverTails). These results confirm the effectiveness of integrating chain-of-thought reasoning with RAG in enhancing the performance, interpretability, and adaptability of guardrail models. With its compact 1.5B parameter architecture, SCoT Guard enables direct deployment on resource-limited IoT edge devices, facilitating real-time local inference while eliminating cloud dependency and ensuring data privacy. Both the training dataset and model weights will be open-sourced.