Towards a Data Provenance Collection and Visualization Framework for Monitoring and Analyzing HPC Environments

Today, the High Performance Computing (HPC) landscape is growing and evolving rapidly and is catalyzing innovations and discoveries to new levels of power and fidelity in a multitude of domains. With this rapid evolution and integration of heterogeneous architectures, technologies and software models underlying HPC infrastructure, managing the confidentiality, integrity, reliability and availability of these complex systems has become extremely challenging. Thus, many research efforts are focusing on monitoring solutions for collecting, correlating and analyzing health metrics and events data for achieving HPC facilities operational efficiency. Data provenance and its sources (such as metadata, lineage) empowers monitoring and event management solutions with capabilities of verification and historical evidence for identifying and troubleshooting HPC infrastructure issues and for forecasting and gaining meaningful insights about the data transformations. In this preliminary work, we present a data provenance collection and visualization infrastructure integrated with the Operations Monitoring and Notification Infrastructure (OMNI) data warehouse at Lawrence Berkeley National Laboratory’s (LBNL) National Energy Scientific Computing Center (NERSC) that has implemented Apache Hop, Neo4j and Grafana Loki for automated root cause analysis and visualization in the face of a computational center’s and users’ critical needs. Moreover, herein we also present several cases illustrating the benefits of employing the proposed framework using 600K concatenated JSON system events and health metrics datasets originating from the Perlmutter computational system, to aid in collection, analysis and visualization of the gathered data provenance. We further elucidate the advantages of this preliminary framework towards reducing the time for detecting and responding to computational center critical issues caused due to physical and cyber threats.