Auto-Detection of Field-Level Dependencies in Data Workflow on a Distributed Platform

With the observed significant rise in the use of data across a variety of industries in the modern world, distributed systems are now required to process and consume Big Data. HPCC (High-Performance Computing Clusters) system is an open-source data lake platform built for high-speed large-volume data engineering. Enterprise Control Language (ECL) is a declarative language specifically designed for huge data projects on the HPCC system platform. Large amounts of data are processed on a regular basis using HPCC systems, In the proposed work an approach to understand and interpret the data flow within an ECL program is investigated. The current system renders an XML graph, which shows the operations at dataset level which can be viewed in the ECL Watch, an interactive web application developed by HPCC systems. As the data changes of individual fields within dataset are not represented, the proposed work field level data and dependencies within datasets are tracked and their changes and operations are visualized as a directed acyclic graph for a generic ECL program to understand its data workflow. The core of this project relies on parsing the ECL IR (Intermediate Representation) emitted by the ECL compiler. The IR generated is transformed into graphical format. The system was tested against sample ECL programs available in ECL watch and other programs available in the platform regression tests and it provided a simple easy to comprehend data flow visualization.