Controllable generation is a crucial technique to provide generated content that meet specific demands in practical applications. An emerging trend for this is utilizing latent variables to represent those demands and guide the generation process, such as with diffusion models. While diffusion models have made significant strides in the realm of image synthesis, applications of diffusion models in text generation remain limited. Specifically, current control signals applied to diffusion language models, such as fixed class labels and abstract syntax trees, are inadequate to capture arbitrary user intents or too abstract for most users to understand and manipulate, making it challenging to meet varied user needs. To tackle this challenge, we propose methods to automatically extract and evaluate fine-grained information in forms of keywords from raw text; on that basis, we encode those keywords into latent variables, and steer latent language models to generate text conditioned on those keywords. To ensure better generalization ability under diverse keywords, we regularize the keyword latent variables through von Mises-Fisher kernel and Principal Component Analysis. Through experiments we demonstrate that fine-grained information in the form of keywords can effectively enhance the generation quality of corresponding text, as well as the effectiveness of our keywords evaluation and regularization methods.