This paper introduces CHEER, an algorithm specifically designed for decentralized nonconvex optimization by leveraging a new design paradigm, which yields better convergence than existing paradigms. In particular, CHEER’s convergence rate scales with 1/T where T is the number of communication rounds, and is less dependent on the underlying communication graphs and compressors. Experimental evaluations validate the superiority of CHEER, demonstrating consistently smaller gradient norms, higher testing accuracy, and significant communication cost savings in the regimes where the communication graph’s spectral gap and the compression rate are small. Importantly, in challenging scenarios where its predecessor encounters convergence difficulties or divergence, CHEER maintains reliable performance. Overall, CHEER offers an efficient and reliable solution for decentralized nonconvex optimization, delivering superior performance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A New Paradigm for Decentralized Nonconvex Optimization with Improved Convergence

  • Tran Thi Phuong,
  • Le Trieu Phong

摘要

This paper introduces CHEER, an algorithm specifically designed for decentralized nonconvex optimization by leveraging a new design paradigm, which yields better convergence than existing paradigms. In particular, CHEER’s convergence rate scales with 1/T where T is the number of communication rounds, and is less dependent on the underlying communication graphs and compressors. Experimental evaluations validate the superiority of CHEER, demonstrating consistently smaller gradient norms, higher testing accuracy, and significant communication cost savings in the regimes where the communication graph’s spectral gap and the compression rate are small. Importantly, in challenging scenarios where its predecessor encounters convergence difficulties or divergence, CHEER maintains reliable performance. Overall, CHEER offers an efficient and reliable solution for decentralized nonconvex optimization, delivering superior performance.