A New Paradigm for Decentralized Nonconvex Optimization with Improved Convergence
摘要
This paper introduces CHEER, an algorithm specifically designed for decentralized nonconvex optimization by leveraging a new design paradigm, which yields better convergence than existing paradigms. In particular, CHEER’s convergence rate scales with 1/T where T is the number of communication rounds, and is less dependent on the underlying communication graphs and compressors. Experimental evaluations validate the superiority of CHEER, demonstrating consistently smaller gradient norms, higher testing accuracy, and significant communication cost savings in the regimes where the communication graph’s spectral gap and the compression rate are small. Importantly, in challenging scenarios where its predecessor encounters convergence difficulties or divergence, CHEER maintains reliable performance. Overall, CHEER offers an efficient and reliable solution for decentralized nonconvex optimization, delivering superior performance.