Scholarlink

Finetuning is an effective method for adapting pretrained networks to downstream tasks. However, the success of finetuning depends heavily on the selection of layers to be tuned, as full finetuning can lead to overfitting, while tuning only the last layer may not capture the necessary task-specific features. This requires a balanced approach of automatic layer selection to achieve higher performance. In this context, we propose the Bias-Variance Guided Layer Selection (BVG-LS), a simple yet effective strategy that adaptively selects a layer to be tuned at each training iteration. More specifically, BVG-LS computes the bias-variance ratios of mini-batch gradients for each layer and updates the parameters of the layer with the largest ratio. This strategy reduces the risk of overfitting while maintaining the model’s capacity to learn task-specific features. In our experiments, we demonstrate the effectiveness of the BVG-LS strategy on seven image classification tasks. We show that BVG-LS outperforms full finetuning on all tasks with the WideEMAs of mini-batch gradients and theResNet-50-2 model and on six out of seven tasks with the ViT-S model (The reproduction code is available online. http://www.vip.sc.e.titech.ac.jp/proj/BVGLS/ ).

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Simple Finetuning Strategy Based on Bias-Variance Ratios of Layer-Wise Gradients

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Mao Tomita,
  • Ikuro Sato,
  • Rei Kawakami,
  • Nakamasa Inoue,
  • Satoshi Ikehata,
  • Masayuki Tanaka

摘要

Finetuning is an effective method for adapting pretrained networks to downstream tasks. However, the success of finetuning depends heavily on the selection of layers to be tuned, as full finetuning can lead to overfitting, while tuning only the last layer may not capture the necessary task-specific features. This requires a balanced approach of automatic layer selection to achieve higher performance. In this context, we propose the Bias-Variance Guided Layer Selection (BVG-LS), a simple yet effective strategy that adaptively selects a layer to be tuned at each training iteration. More specifically, BVG-LS computes the bias-variance ratios of mini-batch gradients for each layer and updates the parameters of the layer with the largest ratio. This strategy reduces the risk of overfitting while maintaining the model’s capacity to learn task-specific features. In our experiments, we demonstrate the effectiveness of the BVG-LS strategy on seven image classification tasks. We show that BVG-LS outperforms full finetuning on all tasks with the WideEMAs of mini-batch gradients and theResNet-50-2 model and on six out of seven tasks with the ViT-S model (The reproduction code is available online. http://www.vip.sc.e.titech.ac.jp/proj/BVGLS/ ).