Scholarlink

Deep neural network (DNN) training systems suffer from communication bottlenecks among workers for gradient synchronization. Gradient compression reduces this overhead but impacts model accuracy, prompting the use of residuals to compensate for the loss. However, we observe that these residuals consume significant GPU memory but fortunately can be reduced with tiny accuracy impact. We propose ResiReduce, a memory-saving mechanism that reuses residuals across similar layers and applies strategic compression within specific layers. Experiments on local and cloud clusters show that ResiReduce can reduce the memory footprint of the model states by up to 15.7% while preserving the model accuracy and training throughput.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Saving Memory via Residual Reduction for DNN Training with Compressed Communication

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Xinjue Zheng,
  • Zhangqiang Ming,
  • Yuchong Hu,
  • Chenxuan Yao,
  • Wenxiang Zhou,
  • Rui Wang,
  • Xun Chen,
  • Dan Feng

摘要

Deep neural network (DNN) training systems suffer from communication bottlenecks among workers for gradient synchronization. Gradient compression reduces this overhead but impacts model accuracy, prompting the use of residuals to compensate for the loss. However, we observe that these residuals consume significant GPU memory but fortunately can be reduced with tiny accuracy impact. We propose ResiReduce, a memory-saving mechanism that reuses residuals across similar layers and applies strategic compression within specific layers. Experiments on local and cloud clusters show that ResiReduce can reduce the memory footprint of the model states by up to 15.7% while preserving the model accuracy and training throughput.