This paper presents a novel method to guide image generation by optimizing the embedding space of diffusion models using evolutionary algorithms. Instead of relying on traditional prompt engineering, the approach directly evolves the prompt embeddings that condition text-to-image generation. Evolutionary operators, such as crossover and mutation, are applied to iteratively refine the embeddings, which are then fed into the diffusion model to generate an image. The fitness of each embedding is determined by the resulting image. Using the SDXL-Turbo model as a test case, a genetic algorithm is employed to optimize its prompt embeddings, leading to improvements in fitness as measured by the LAION Aesthetics Predictor V2. Results show that over generations, the optimized embeddings yield significant gains in fitness scores compared to the initial training images. The underlying framework is publicly available and executable in a Jupyter Notebook, allowing for further experimentation and adaptation to various generative tasks.