Scholarlink

<p>Parameter-efficient transfer learning (PETL) in video action recognition (VAR) effectively mitigates the challenges of transfer learning and facilitates the utilization of reliable prior knowledge from visual-language models. Recently, side-tuning methods have emerged in VAR, significantly reducing memory usage without excessively increasing tunable parameters. However, most existing PETL works in VAR focus solely on either "parameter efficiency" or "memory efficiency". In this paper, we propose a plug-and-play module named temporal difference module for efficient video action recognition (TD4V), which achieves superior performance in both fine-tuning for "parameter efficiency" and side-tuning for "memory efficiency." TD4V explicitly extracts temporal clues through difference operations and utilizes a 2D convolution layer to extract features from the temporal difference information distributed in space. We integrate TD4V into CLIP for both fine-tuning and side-tuning scenarios. In fine-tuning, we freeze all parameters of CLIP and only tune TD4V with minimal parameters. In side-tuning, we design a lightweight CLIP with TD4V as the side network, significantly reducing memory usage by 60.7% (from 34.9G in fine-tuning to 13.7G in side-tuning) through backpropagation that passes only through the lightweight side network. Experimental results demonstrate that TD4V achieves state-of-the-art or comparable performance on benchmark datasets (80.37% on HMDB-51, 96.91% on UCF-101, and 70.18% on SSv2) with significantly fewer parameters (2.14&#xa0;M in fine-tuning and 20.44&#xa0;M in side-tuning) compared to existing methods. Our approach thus offers a practical solution for efficient video action recognition through PETL. The code is available at <a href="https://github.com/owlholy/TD4V">https://github.com/owlholy/TD4V</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

TD4V: temporal difference module for efficient video action recognition via fine-tuning and side-tuning

  • 期刊论文

  • 发表日期：

    2025年12月6日

查看全文
  • Youwei Li,
  • Junyong Ye,
  • Guangyi Xu,
  • Jingjing Wang,
  • Xinyuan Liu

摘要

Parameter-efficient transfer learning (PETL) in video action recognition (VAR) effectively mitigates the challenges of transfer learning and facilitates the utilization of reliable prior knowledge from visual-language models. Recently, side-tuning methods have emerged in VAR, significantly reducing memory usage without excessively increasing tunable parameters. However, most existing PETL works in VAR focus solely on either "parameter efficiency" or "memory efficiency". In this paper, we propose a plug-and-play module named temporal difference module for efficient video action recognition (TD4V), which achieves superior performance in both fine-tuning for "parameter efficiency" and side-tuning for "memory efficiency." TD4V explicitly extracts temporal clues through difference operations and utilizes a 2D convolution layer to extract features from the temporal difference information distributed in space. We integrate TD4V into CLIP for both fine-tuning and side-tuning scenarios. In fine-tuning, we freeze all parameters of CLIP and only tune TD4V with minimal parameters. In side-tuning, we design a lightweight CLIP with TD4V as the side network, significantly reducing memory usage by 60.7% (from 34.9G in fine-tuning to 13.7G in side-tuning) through backpropagation that passes only through the lightweight side network. Experimental results demonstrate that TD4V achieves state-of-the-art or comparable performance on benchmark datasets (80.37% on HMDB-51, 96.91% on UCF-101, and 70.18% on SSv2) with significantly fewer parameters (2.14 M in fine-tuning and 20.44 M in side-tuning) compared to existing methods. Our approach thus offers a practical solution for efficient video action recognition through PETL. The code is available at https://github.com/owlholy/TD4V.