We introduce a new approach in distributed learning, building on Hinton’s Forward-Forward (FF) algorithm to speed up the training of neural networks in distributed environments without losing accuracy. Unlike traditional methods that rely on forward and backward passes, the FF algorithm employs a dual forward pass strategy, eliminating the dependency among layers required during the backpropagation period, which prevents efficient parallelization of the training process. Although the original FF algorithm focused on its ability to match the performance of the backpropagation algorithm, this work aims to reduce the training time with pipeline parallelism. We propose three novel pipelined FF algorithms that speed up training 3.75 times on the MNIST dataset while maintaining accuracy when training a four-layer network with four compute nodes. These results show that FF is highly parallelizable and its potential in large-scale distributed/federated systems to enable faster training for larger and more complex models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Going Forward-Forward in Distributed Deep Learning

  • Ege Aktemur,
  • Ege Zorlutuna,
  • Kaan Bilgili,
  • Tacettin Emre Bök,
  • Berrin Yanikoglu,
  • Süha Orhun Mutluergil

摘要

We introduce a new approach in distributed learning, building on Hinton’s Forward-Forward (FF) algorithm to speed up the training of neural networks in distributed environments without losing accuracy. Unlike traditional methods that rely on forward and backward passes, the FF algorithm employs a dual forward pass strategy, eliminating the dependency among layers required during the backpropagation period, which prevents efficient parallelization of the training process. Although the original FF algorithm focused on its ability to match the performance of the backpropagation algorithm, this work aims to reduce the training time with pipeline parallelism. We propose three novel pipelined FF algorithms that speed up training 3.75 times on the MNIST dataset while maintaining accuracy when training a four-layer network with four compute nodes. These results show that FF is highly parallelizable and its potential in large-scale distributed/federated systems to enable faster training for larger and more complex models.