Large vision-language models (LVLMs), such as CLIP and Stable Diffusion, exhibit remarkable versatility in diverse applications but remain sensitive to poorly formed or ambiguous user queries. This sensitivity often results in suboptimal outputs, limiting their effectiveness in real-world scenarios. To address this challenge, we propose a novel approach that refines the user queries to generate query-specific images. This work explores the integration of CLIP (Contrastive Language-Image Pretraining) and Stable Diffusion for generating high-quality, task-specific images from both generic and task-specific text prompts. We propose a novel framework where task-specific (generated using large language model)and generic prompts are encoded through CLIP’s text encoder, and task-specific images are encoded through CLIP’s vision encoder. To align these diverse embeddings, we employ contrastive learning, which optimizes the proximity of text embeddings to ensure that task-specific textual descriptions are coherently aligned with generic prompts. After the contrastive alignment, a text decoder is used to convert the optimized embeddings into natural language descriptions, which are then fed into Stable Diffusion for image generation. This end-to-end pipeline facilitates the creation of diverse and contextually accurate images, enabling conditional image generation based on both generic and task-specific textual inputs. Our approach demonstrates significant promise in enhancing image generation tasks where text and image modalities are tightly coupled, offering an efficient solution for high-quality, context-aware image synthesis. We release code at https://github.com/s4nyam/isitlargeenough .