Scholarlink

<p>Recently, <i>Multi-modal Large Language Models</i> (MLLMs) have garnered an influx of interest from both academia and industry. Despite great progress, MLLMs still have a high demand of downstream task tuning for applications, which consumes excessive parameter and computation overhead. In this paper, we propose a novel parameter and computation efficient tuning method for MLLMs, termed <i>Effective Attention Skipping</i> (EAS). Concretely, we first reveal that <i>multi-head attentions</i> (MHAs) in MLLMs, the primary source of computation, are often redundant to downstream tasks. Based on this observation, EAS evaluates attention redundancy and skips the less important MHAs to speed up inference. Besides, we also propose a novel <i>propagation-of-information adapter</i> (PIA) to serve attention skipping while maintaining parameter efficiency. More importantly, PIA can be further re-parameterized into <i>feed-forward networks</i> (FFNs) for zero-extra latency. To validate EAS, we apply it to three common MLLMs, and conduct extensive experiments on six downstream tasks. The experimental results show that EAS can not only retain the high performance of MLLMs but also reduce the updated parameters scale greatly, while speeding up inference speed to a large extent. For instance, LLaVA-EAS can obtain 96.2% accuracy while accelerating the inference speed by about +20% on ScienceQA. Our code is publicly released at <a href="https://github.com/DoubtedSteam/EAS">https://github.com/DoubtedSteam/EAS</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Not All Attention is Needed: Parameter and Computation Efficient Tuning for Multi-modal Large Language Models via Effective Attention Skipping

  • 期刊论文

  • 发表日期：

    2026年2月21日

查看全文
  • Qiong Wu,
  • Yiyi Zhou,
  • Weihao Ye,
  • Xiaoshuai Sun,
  • Rongrong Ji

摘要

Recently, Multi-modal Large Language Models (MLLMs) have garnered an influx of interest from both academia and industry. Despite great progress, MLLMs still have a high demand of downstream task tuning for applications, which consumes excessive parameter and computation overhead. In this paper, we propose a novel parameter and computation efficient tuning method for MLLMs, termed Effective Attention Skipping (EAS). Concretely, we first reveal that multi-head attentions (MHAs) in MLLMs, the primary source of computation, are often redundant to downstream tasks. Based on this observation, EAS evaluates attention redundancy and skips the less important MHAs to speed up inference. Besides, we also propose a novel propagation-of-information adapter (PIA) to serve attention skipping while maintaining parameter efficiency. More importantly, PIA can be further re-parameterized into feed-forward networks (FFNs) for zero-extra latency. To validate EAS, we apply it to three common MLLMs, and conduct extensive experiments on six downstream tasks. The experimental results show that EAS can not only retain the high performance of MLLMs but also reduce the updated parameters scale greatly, while speeding up inference speed to a large extent. For instance, LLaVA-EAS can obtain 96.2% accuracy while accelerating the inference speed by about +20% on ScienceQA. Our code is publicly released at https://github.com/DoubtedSteam/EAS.