Recent advances in artificial intelligence (AI) text generation systems have resulted in their ability to provide precise recommendations in response to users’ questions (prompts). However AI models often operate as black boxes, making it challenging for users to comprehend their inner workings. The transparency of these models is crucial for users to gain a better understanding of how AI systems function. While the Human-Computer Interaction (HCI) community has advocated for design principles like progressive disclosure to improve transparency, we still lack empirical evidence validating its efficacy for AI systems, especially in the context of LLM-based text generation. Addressing this gap, this paper presents a user study with 30 participants aimed at investigating the effect of progressive disclosure and adjusting the explanations so as to adapt to users’ mental models for improving the transparency of AI text generation systems. The findings suggest that users prefer on-demand explanations and value diverse explanation methods, especially when the explanations gradually give the users a better understanding of the AI system. Additionally, qualitative data shows a marginal preference for word clouds over keyword highlighting. User feedback indicates that explanations such as word-pair cosine values, which leverage the interpretability of AI models, are less suitable for lay users. Altering the visual presentation of these word-pair cosine values from a table of numbers to a bar graph did not increase user satisfaction with this explanation technique.