Recent text-to-image generative models, such as stable diffusion models (SDs), have achieved remarkable performance. However, writing a prompt that precisely aligns with user preference is challenging as it requires expert knowledge for a given SD, where different SDs fine-tuned on different sets of stylized images can have different prompt templates. In this paper, we introduce HYTI, a Hyperbolic Text-to-Image model which streamlines the prompt-writing process in an easy hierarchical manner. Specifically, our motivation is built upon the fact that in hyperbolic models like the Poincaré disk and the Lorentz model, geodesics appear to curve away from the original point, allowing for an efficient representation of tree-like structures. With this insight, we first add the new attribute tokens in the original CLIP text encoder, and then fine-tune their text embeddings into hyperbolic space under Lorentz model, a hyperbolic space with a time axis. With extensive experiments on CelebA, FAIR, FairFace and LHQ, we demonstrate that HYTI obtains better image quality over attribute control than stable diffusion baselines. Moreover, the text embedding from HYTI achieves better text-image alignment, indicating the advantage of using hyperbolic space over Euclidean space for hierarchical attribute control.