Scholarlink

<p>Fine-tuning pre-trained language models (PLMs) is crucial for achieving performance gains in downstream natural language processing tasks, but it is prone to overfitting in low-resource scenarios. Existing noise-based regularization methods typically rely on static parameter statistics or predefined heuristics to mitigate this issue. However, the noise magnitude and injection timing in these approaches depend on fixed hyperparameters or heuristic schedules, making them unable to adapt to the dynamic optimization state of the model during training. To address this limitation, this paper proposes a novel fine-tuning framework named gradient-guided noise injection (GNI). The core idea is to utilize gradient information generated during optimization to dynamically adjust the noise intensity in a layer-wise and real-time manner. Extensive experiments on multiple tasks from the GLUE and SuperGLUE benchmarks demonstrate that GNI generally improves average performance across different pre-trained models and task settings, while exhibiting stable generalizability and robustness. This work provides a simple yet effective solution for optimization-state aware dynamic regularization.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Gradient-guided layerwise adaptive noise injection for pre-trained language model fine-tuning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月1日

查看全文
  • Qinglin Jiang,
  • Cheng Zeng,
  • Nan Chi

摘要

Fine-tuning pre-trained language models (PLMs) is crucial for achieving performance gains in downstream natural language processing tasks, but it is prone to overfitting in low-resource scenarios. Existing noise-based regularization methods typically rely on static parameter statistics or predefined heuristics to mitigate this issue. However, the noise magnitude and injection timing in these approaches depend on fixed hyperparameters or heuristic schedules, making them unable to adapt to the dynamic optimization state of the model during training. To address this limitation, this paper proposes a novel fine-tuning framework named gradient-guided noise injection (GNI). The core idea is to utilize gradient information generated during optimization to dynamically adjust the noise intensity in a layer-wise and real-time manner. Extensive experiments on multiple tasks from the GLUE and SuperGLUE benchmarks demonstrate that GNI generally improves average performance across different pre-trained models and task settings, while exhibiting stable generalizability and robustness. This work provides a simple yet effective solution for optimization-state aware dynamic regularization.