Generalized Hermite Processes (GHP) have been used to model financial systems analytically. Nevertheless, their potential uses in machine learning applications have been less studied. By varying the parameters of the GHP, mainly the Hurst exponent and the degree of the Hermite polynomial, different realization paths can be simulated using software exhibiting different roughness and memory patterns. We sample these paths and pre-train a LSTM neural network on them for different numbers of epochs. Then, the model is presented with domain data in three contexts: zero-shot, one-shot and fine-tuned on the domain data for 10 epochs, and the fit is assessed using sequential 5-fold cross-validation for three different financial time series data sets with different provenance and characteristics. The approach is promising as the pretrained models show surprisingly good fit even for the zero-shot context. For two of the datasets, the approach outperforms the baseline model trained directly on the domain data by a significant margin, and further fine-tuning improves fit up to a point, while on the third dataset the reference model outperforms the proposed approach by a small margin, with further fine-tuning unable to close the error gap. The experiments show the applicability of using synthetic data obtained from GHP to pre-train neural networks for time series regression problems.