Scholarlink

<p>Parameter-efficient transfer learning (PETL) is an emerging research spot aimed at inexpensively adapting large-scale pre-trained models to downstream tasks. Despite the effectiveness, we notice that most existing PETL methods usually increase the complexity of already redundant pre-trained models, thus greatly limiting their real-world application. In this paper, we introduce a more practical setting of PETL called <i>parameter-efficient network pruning</i> (PENP), which aims to realize the network sparsification during PETL. However, common <i>network pruning</i> (NP) methods usually require tuning the entire model, thus struggling to directly combine with PETL. To address this issue, we theoretically find that the re-parameterized adapter is essential to bridge the gap of NP and PETL. Motivated by this, we design a novel PETL module for vision models, namely RepAdapter. Different from common adapters, RepAdapter removes the non-linear function to realize the re-parameterization. To maximize its benefit to vision models, we also carefully study the design of its structure and placement. Based on RepAdapter, we propose an innovative optimization strategy for PENP, namely <i>sparse-aware adapter tuning</i> (SA-Tuning), which encourages the network sparsity via a parameter division strategy and a penalty loss. To validate RepAdapter and SA-Tuning, we conduct extensive experiments on 22 benchmark datasets of three vision tasks, <i>i.e.,</i> image and video classifications and semantic segmentation. Experimental results show the superior performance and efficiency of our method than existing PETL methods and three pruning baselines. For instance, RepAdapter outperforms full tuning by +7.2% on average and saves up to 25% training time, 20% GPU memory, 30% GFlops and 94.6% storage cost of ViT-B/16 on VTAB-1k. The generalization ability of our method is also well validated by a bunch of vision models, <i>i.e.</i>, ViT, Swin-Transformer and ConvNeXt.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Towards Parameter-Efficient Network Pruning with Re-Parameterized Adapter

  • 期刊论文

  • 发表日期：

    2026年3月9日

查看全文
  • Gen Luo,
  • Yuxin Zhang,
  • Yiyi Zhou,
  • Minglang Huang,
  • Xiaoshuai Sun,
  • Rongrong Ji

摘要

Parameter-efficient transfer learning (PETL) is an emerging research spot aimed at inexpensively adapting large-scale pre-trained models to downstream tasks. Despite the effectiveness, we notice that most existing PETL methods usually increase the complexity of already redundant pre-trained models, thus greatly limiting their real-world application. In this paper, we introduce a more practical setting of PETL called parameter-efficient network pruning (PENP), which aims to realize the network sparsification during PETL. However, common network pruning (NP) methods usually require tuning the entire model, thus struggling to directly combine with PETL. To address this issue, we theoretically find that the re-parameterized adapter is essential to bridge the gap of NP and PETL. Motivated by this, we design a novel PETL module for vision models, namely RepAdapter. Different from common adapters, RepAdapter removes the non-linear function to realize the re-parameterization. To maximize its benefit to vision models, we also carefully study the design of its structure and placement. Based on RepAdapter, we propose an innovative optimization strategy for PENP, namely sparse-aware adapter tuning (SA-Tuning), which encourages the network sparsity via a parameter division strategy and a penalty loss. To validate RepAdapter and SA-Tuning, we conduct extensive experiments on 22 benchmark datasets of three vision tasks, i.e., image and video classifications and semantic segmentation. Experimental results show the superior performance and efficiency of our method than existing PETL methods and three pruning baselines. For instance, RepAdapter outperforms full tuning by +7.2% on average and saves up to 25% training time, 20% GPU memory, 30% GFlops and 94.6% storage cost of ViT-B/16 on VTAB-1k. The generalization ability of our method is also well validated by a bunch of vision models, i.e., ViT, Swin-Transformer and ConvNeXt.