Tool Augmented LLMs for Big Data Analysis

Recent advancements in artificial intelligence, have been driven by the development of Large Language Models (LLMs). Following the emergence of these models, the field has witnessed remarkable progress, paving the way for unprecedented innovations. Despite these advancements, the application of LLMs in real-world systems faces challenges due to the diverse range of tools and environments, necessitating innovative approaches for practical deployment. A critical focus has been on enhancing LLMs’ text-oriented question-answering capabilities through interaction with tools and APIs, aiming to reduce hallucinations and boost performance. In this paper we explore LLMs’ capability for handling and learning tools for Big Data Analysis. We propose a pipeline and a methodology for developing LLMs that can interface with complex big data environments. By achieving this, we significantly lower the barriers to entry for non-expert users, enabling them to leverage big data analytics and visualization tools without requiring prior knowledge of the database type or the tools’ implementation details.