Management of Data Flows Between Cloud, HPC and IoT/Edge

The components of heterogeneous applications are deployed across various execution platforms and utilise the capabilities of the platforms. As such, one component can utilise HPC resources for better performance in batch computations, while another—Cloud resources, for better scalability and elasticity. Furthermore, there is also a possibility for processing on Edge devices. The usage of such a hybrid setup, where dependent components of the applications are deployed across various platforms, might require flexible and adaptive data transfers from one platform to another. This work presents a data management framework, based on the Apache NiFi dataflow management system and developed in the scope of the SODALITE EU project. This framework enables scalable data transfer between any of GridFTP (a file transfer protocol common in HPC), HTTP, S3-compatible and data streaming (such as MQTT) endpoints.