Traditional recurrent neural networks (RNN) suffer from issues in learning long-term dependencies since the hidden layer is computed using the hidden layer outputs at past and current moments. To improve the model’s ability to model long-term dependencies, this paper presents a model based on second-order RNN, which hidden layer is computed based on the input at the current moment and the outputs of the hidden layer at the previous two moments. The model focuses on the problem of obtaining long-term dependencies for traditional RNN. In this work, the effectiveness of the model is confirmed on twelve datasets using a second-order RNN model, or SndRNN, which was built using a time series forecasting task book. The experiments we conducted on twelve datasets demonstrate that the SndRNN yield better performance than VanillaRNN by 32.64% and 1.33% on average in terms of RMSE and R2.