Effizienter Datenmanagement-Workflow zur Analyse eines PtGtX-Systems basierend auf Open-Source-Software und Low-Cost-Hardware

In order to efficiently optimize a power-to-gas-to‑X system, a digital twin can be created as a simulation model based on experimental data for a laboratory system and modified accordingly. In addition, data from experiment and simulation must be continuously recorded and processed for monitoring and online simulation. Overall, an efficient data management workflow is required.

In this paper, a workflow consisting of free, established and scalable open-source software is outlined for the application at hand and, in particular, a suitable data model is developed, implemented and its resource-saving realization on low-cost hardware is demonstrated. Depending on the data modeling, inexpensive and old hardware may be sufficient for the required task.

Apache NiFi is used to create a visual workflow for retrieving and processing data from various sources. The extracted data is aggregated in Apache Cassandra, a data system widely used for its performance, scalability and durability.

Grafana is used for visual monitoring of the system. The entire system is built using Docker containers for the purpose of reproducibility and efficient deployment.

Benchmarks and realistic hardware and data modeling configurations demonstrate the stated performance and sustainability of the proposed solution.