Batch Level Distributed Training of LSTM with Infinity Norm Gradient Flow

The advent of the big data era has led to a substantial increase in available data for analysis and prediction, creating a need for effective utilization of this vast input to improve prediction quality. LSTM-based neural networks have demonstrated exceptional performance in tasks such as time series forecasting. However, the effectiveness of these models can be constrained by the limitations of GPU memory. Distributed computing has emerged as a promising solution to address the challenges posed by large-sample, long-sequence time series forecasting. This work develops a novel distributed training method for LSTM-based time series forecasting under big data scenario. Infinity norm gradient flow (INGF) is applied to speed up the convergence, acceleration techniques are designed to improve the utility rate of multiple GPUs. The study showcases significant insights into the performance of various distributed strategies and optimization techniques for batch level distributed training. As a result, we achieve an impressive tenfold increase in efficiency while making only a negligible sacrifice in accuracy.