Scholarlink

Vision Transformers (ViTs) perform well in computer vision but need high computational power. This paper explores Grouped Query Attention (GQA) to reduce parameters in TinyViT models with competitive accuracy. We compare GQA settings with query-to-key ratios of 2:1 to 10:1 on CIFAR-10 and CIFAR-100 benchmarks. Findings show that GQA decreases attention parameters by a maximum of 42.8% with no loss of accuracy. The GQA 5:1 setup reaches 89.95% accuracy in CIFAR-10, surpassing the usual MHA’s 89.37% while shrinking attention parameters by 39.9%. These results prove GQA’s viability for deploying effective vision transformers within resource-limited environments.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Compact Yet Powerful: Group Query Attention in TinyViT Student Models for Efficient Classifications

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Nishitha Anand,
  • Rachit Verma,
  • Bhaskarjyoti Das

摘要

Vision Transformers (ViTs) perform well in computer vision but need high computational power. This paper explores Grouped Query Attention (GQA) to reduce parameters in TinyViT models with competitive accuracy. We compare GQA settings with query-to-key ratios of 2:1 to 10:1 on CIFAR-10 and CIFAR-100 benchmarks. Findings show that GQA decreases attention parameters by a maximum of 42.8% with no loss of accuracy. The GQA 5:1 setup reaches 89.95% accuracy in CIFAR-10, surpassing the usual MHA’s 89.37% while shrinking attention parameters by 39.9%. These results prove GQA’s viability for deploying effective vision transformers within resource-limited environments.