The paper discusses the integration of Generative AI, specifically ChatGPT, in Clinical Decision Support Systems (CDSS). These intelligent models have applications that are being widely adopted in every industry, even in a clinical setting. It helps optimize Clinical Decision Support (CDS) and provides unique suggestions. It can recognize possible modifications to alert logic and even help clinicians recommend CDS development. However, AI use does have certain risks, such as hallucination. ChatGPT, a very well-known generative AI model, can handle large amounts of data, and it also makes it convenient for clinicians to review diagnostic suggestions during the process of CDS. This paper examines the possible challenges of implementing ChatGPT in a clinical setting. Our research introduces a new model, essentially an AI loop called Spiral Human-In-The-Generative (Spiral HITL), which seeks to mitigate risks involved with AI applications in CDS.