We analyze the impact of structure preservation inside the topology of a neural network on the error of the neural network. To do this we split the error into an approximation, optimization and generalization term. The analysis is of empirical nature and is performed on the example of preserving the symplectic structure of Hamiltonian systems. For this, three different types of neural networks are trained on data from two different Hamiltonian systems. We find that the main improvement of structure-preserving neural networks comes in the form of a greatly reduced generalization error.