Currently, large language models are actively developing and beginning to be used to solve some mathematical problems. With the emergence of xLSTM model, which demonstrates the results comparable with transformer-based models, there has been a surge of interest in recurrent neural networks. This paper considers the application of baseline recurrent models such as LSTM and GRU for solving several types of differential equations. In this paper, differential equations are considered as text sequences, and LSTM and GRU models are applied to them to translate the text ‘equation’ into the text ‘solution’. The quality of the models is assessed using the BLEU machine translation quality metric. In this work, two datasets were collected for fine-tuning the considered models. First, a dataset of 1,054 equation-solution pairs was obtained from reference textbooks. Second, a synthetic dataset of 9,548 equation-solution pairs, containing linear homogeneous differential equations of the second and the third order, was generated. Both datasets are publicly available and can be used by researchers to fine-tune different large language models to solve differential equations. Our computer experiments have shown that fine-tuning models on the dataset of 1,054 equation-solution pairs led to very low BLEU scores (0.16 and less), while fine-tuning on the larger synthetic dataset increased the BLEU scores. For the synthetic dataset, the best result was achieved by the GRU model with a BLEU score of 0.569. A possible reason for these moderate BLEU scores is the limitations on the model parameters, namely, a relatively small number of layers (no more than four) and a limited number of neurons (no more than 256), which can be not enough for the considered problem.