Open government data has been in the spotlight for several years, emphasising the need for publicly available datasets to foster societal innovation and enable data-driven decision-making. However, it is not sufficient for data to be available through an open data portal or repository to be usable by end users and for them to fully unveil its potential; it also needs to be accessible and intelligible. Targeting this aspect, this study builds on the synergy of open government data and emerging technologies, such as Large Language Models (LLMs) to demonstrate that technologically enabled open data portals can have enormous possibilities for end-users to understand an available dataset efficiently, and in an interactive manner. This implementation is built on the LLMs Retrieval Augmented Generation (RAG) functionality which fetches datasets after a user’s choice from the Greek Open Data Portal, allowing the user to ask questions about this dataset using natural language. The Greek open data portal provides access to datasets from different domains through the Application Programming Interfaces (APIs) which makes it easier to bridge the gap between complex data and non-technical users who want to use it. This work explores the implementation potential of such a pipeline in a real-world application using the Greek Open Data portal as data source, and by utilising a conversational smart agent to interact with the open datasets available through the portal, it brings forward new capabilities in data-user interaction and efficient data exploration. Apart from the implementation, this study explores the pitfalls and shortcomings of such an endeavour (conversational agents) which requires real-time calculations and deterministic responses.