This paper investigates the less explored domain of image-based plant growth modeling using deep learning. Our approach aims to model plant phenotypes at different growth stages, laying the foundation for further research on generating synthetic training data for object detection and segmentation. We introduce a novel class of neural network architectures called Latent Plant Growth Models (LPGMs), which model the temporal aspects of plant growth within the latent space of pre-trained autoencoders. Two distinct LPGM architectures have been developed and investigated. These architectures demonstrate improved performance over existing state-of-the-art methods in terms of image quality and training time, while retaining a high degree of realism in the simulated growth.