Recent advances in image generative models have enabled rich collaborations between humans and AI systems. Among these, Energy-Based Models (EBMs) learn an energy landscape that guides noised samples toward high-probability regions. Unlike diffusion models that use fixed time schedules, EBMs possess equilibrium properties that enable user feedback during generation without destabilizing the distribution. However, current EBM research primarily optimizes for high-fidelity images, offering little control over the trade-off between semantic realism and fine-grained diversity—an essential feature for interactive creative applications. Artists and creatives thus lack a modality to balance semantic coherence (e.g., “a red apple”) with creative variation (e.g., apples of different shapes or colors). To address this, we introduce a geometry-aware annealing framework for EBMs. We propose a directionally-aware annealing variable that leverages local geometric information to directionally adjust the effective noise level during sampling. Such an annealing feedback mechanism that allows users to explore higher-diversity sample contenders before selectively generating realistic images from chosen contenders. Together, these techniques enable a controllable balance between fidelity and creativity, advancing the use of EBMs for interactive creative AI.