Video captioning aims to generate textual descriptions for given videos. Transformer architectures have been widely adopted in video captioning tasks due to their ability to simultaneously model spatio-temporal features in videos and language information. However, existing methods typically rely on the standard Transformer design, where the core components, Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN), are arranged in a fixed structure. This rigid design fails to adapt to the varying functional roles and hierarchical demands of different layers, leading to inefficient parameter utilization and limited performance in terms of inference speed and resource efficiency. To address these limitations, we introduce FlexTransformer, a lightweight vision-language model that employs a flexible module allocation strategy to achieve efficient video captioning. Specifically, we adaptively redesign the ratio between FFN and MHSA across layers, enabling the lower layers to focus on fine-grained local spatio-temporal feature extraction, while the higher layers emphasize global semantic modeling. In addition, we compress the internal channel dimensions of Q and K in MHSA to half of the embedding dimension and reduce the FFN expansion ratio from the conventional \(4{\times} \) to \(2{\times} \) . Experimental results on two mainstream video captioning benchmark datasets MSVD and MSRVTT show that our proposed approach greatly reduces 73.9% Flops via pruning 53.4% parameters compared with the base model while achieving competitive performance with SOTA methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

FlexTransformer: Flexibly Allocating Transformer Modules for Efficient Video Captioning

  • Qianwen Cao,
  • Hangyuan Shi,
  • Gongru Li,
  • Che Li

摘要

Video captioning aims to generate textual descriptions for given videos. Transformer architectures have been widely adopted in video captioning tasks due to their ability to simultaneously model spatio-temporal features in videos and language information. However, existing methods typically rely on the standard Transformer design, where the core components, Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN), are arranged in a fixed structure. This rigid design fails to adapt to the varying functional roles and hierarchical demands of different layers, leading to inefficient parameter utilization and limited performance in terms of inference speed and resource efficiency. To address these limitations, we introduce FlexTransformer, a lightweight vision-language model that employs a flexible module allocation strategy to achieve efficient video captioning. Specifically, we adaptively redesign the ratio between FFN and MHSA across layers, enabling the lower layers to focus on fine-grained local spatio-temporal feature extraction, while the higher layers emphasize global semantic modeling. In addition, we compress the internal channel dimensions of Q and K in MHSA to half of the embedding dimension and reduce the FFN expansion ratio from the conventional \(4{\times} \) to \(2{\times} \) . Experimental results on two mainstream video captioning benchmark datasets MSVD and MSRVTT show that our proposed approach greatly reduces 73.9% Flops via pruning 53.4% parameters compared with the base model while achieving competitive performance with SOTA methods.