A large language model (LLM) is an artificial intelligence (AI) model designed to process and generate responses in human language. Nowadays, LLMs are playing an increasingly vital role in a wide range of applications, including process industries, medical, finance, management, and infrastructure. However, LLMs may produce inaccurate information; therefore, it is essential to check the reliability of the responses from LLMs before using them for the intended purpose. This study presents a novel approach to verify the reliability of the interpretation of drinking water quality parameters by open-source LLM Gemini 1.5 Flash, Gemini 1.5 Pro (Advanced), and ChatGPT-4o. The data used in the study is collected in real-time from an IoT-enabled smart water distribution system at Indian Institute of Technology Jodhpur. Monitoring vital quality parameters such as chlorine, pH, total dissolved solids (TDS), and temperature demonstrates acceptable levels, ensuring compliance with safety standards. The purpose of the chapter is to attempt to check the reliability of the insights and recommendations generated by LLM and provide a case for its usefulness by validating it from a dataset obtained from a real-world case and close the knowledge gap between general awareness and the potential of LLM-integrated smart water grids in sustainable management of water distribution systems (WDSs).