Model Variance in Neural Network Models for Forecasting Chaotic Systems

While the non-linearity and sensitivity of chaotic systems make the forecasting of their future behaviour challenging, the ability of machine learning techniques to accurately represent complex dynamics has placed them at the forefront of this field. However, small differences in the model used can lead to noticeable differences in observable model performance due to the systems’ sensitivity. This can result in even seemingly inconsequential changes to the training process causing practical differences in model performance. In this study performance metrics for both next-step prediction accuracy and a novel metric for the duration of accurate prediction have been calculated for a number of long-short term memory (LSTM) models, and their variance studied. The results highlight the causes and quantifies the scale of inconsistency in model performance that can be found during seemingly equivalent training scenarios, and concludes that such variance in model performance is non-trivial.