Prompt-based image generation models have shown remarkable advances in recent years, enabling users to generate high-quality images from natural language descriptions. However, the lack of transparency of these models often leaves users uncertain about how specific prompts influence the generated outputs. In this paper, we propose a novel approach to integrating explainability into prompt-based image generation, aiming to develop transparent generative models that offer users more profound insights into the decision-making processes of these systems. We introduce concept attribution methods to provide clear, interpretable feedback on translating prompts into visual content. In addition, we propose a comprehensive evaluation framework to assess the effectiveness of these explainability features and the quality of the generated image from a user-generated text description. With our framework, explainability-enhanced models should maintain high-quality image synthesis and significantly improve user trust and understanding. This research contributes to the broader field of explainable AI (XAI) by advancing methods that make complex generative models more accessible and accountable to users, paving the way for more responsible and user-centric AI technologies.