Scholarlink

<p>Quantization-aware training (QAT) has become a crucial technique for compressing large language models (LLMs), enabling low-bit deployment in resource-constrained environments. However, existing QAT methods often treat blocks in isolation, neglecting structural dependencies between adjacent blocks. This can lead to severe accuracy degradation and inefficient training, especially under ultra-low-bit quantization. To address these limitations, we propose Block-wise Shared Quantization-Aware Training (BSQAT), a two-stage framework that combines local and global optimization for efficient and accurate model quantization. In the Local QAT (LQAT) phase, BSQAT merges adjacent Transformer blocks using an overlapping sliding-window strategy and assigns shared quantization parameters to capture inter-block correlations. To preserve fine-grained weight fidelity, a lightweight low-rank scaling module is designed within each merged block. In the Global QAT (GQAT) phase, global scaling parameters are fine-tuned with minimal overhead to enhance overall model robustness. Extensive experiments demonstrate that BSQAT generalizes well across multiple model architectures including LLaMa, OPT, Vicuna, and Mistral, and scales effectively from 6.7B to 123B parameters. In INT2 quantization of the LLaMa-3-70B model, BSQAT improves average zero-shot accuracy from 65.18% to 67.87% compared to EfficientQAT, while reducing memory usage by approximately 7%.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Bsqat: block-wise shared quantization-aware training for large language models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月17日

查看全文
  • Senbao Hou,
  • Libin Hou,
  • Tianyuan Liu,
  • Linyuan Wang,
  • Bin Yan

摘要

Quantization-aware training (QAT) has become a crucial technique for compressing large language models (LLMs), enabling low-bit deployment in resource-constrained environments. However, existing QAT methods often treat blocks in isolation, neglecting structural dependencies between adjacent blocks. This can lead to severe accuracy degradation and inefficient training, especially under ultra-low-bit quantization. To address these limitations, we propose Block-wise Shared Quantization-Aware Training (BSQAT), a two-stage framework that combines local and global optimization for efficient and accurate model quantization. In the Local QAT (LQAT) phase, BSQAT merges adjacent Transformer blocks using an overlapping sliding-window strategy and assigns shared quantization parameters to capture inter-block correlations. To preserve fine-grained weight fidelity, a lightweight low-rank scaling module is designed within each merged block. In the Global QAT (GQAT) phase, global scaling parameters are fine-tuned with minimal overhead to enhance overall model robustness. Extensive experiments demonstrate that BSQAT generalizes well across multiple model architectures including LLaMa, OPT, Vicuna, and Mistral, and scales effectively from 6.7B to 123B parameters. In INT2 quantization of the LLaMa-3-70B model, BSQAT improves average zero-shot accuracy from 65.18% to 67.87% compared to EfficientQAT, while reducing memory usage by approximately 7%.