Accurate load forecasting is essential for efficient grid management, contributing to energy stability and the integration of renewable resources. Recurrent Neural Networks (RNNs) have become particularly popular due to their ability to capture temporal dependencies in load data. However, the optimal RNN architecture and hyperparameter choices for achieving both high accuracy and computational efficiency remain an open question. In this paper, we compare the performance of Long Short-Term Memorys (LSTMs), Gated Recurrent Units (GRUs), State Space Models (SSMs), MinLSTM, and MinGRU models across different hyperparameter settings ranging from 14 to around 365k parameters and test their performance on simulated load data for two Low-Voltage grid topologies. Results show that the LSTM performs best across all tested models. Furthermore, we observe that the benefits of increasing model size for the tested RNNs plateau relatively quickly after around 5k parameters. For further increasing the parameter efficiency of trained SSM models, we apply model order reduction via balanced truncation. This allows for reducing the parameter count of trained SSM models by up to 79% without any need for retraining, while maintaining the same level of accuracy and, in some cases, even improves the test loss by up to 6%.