In the realm of big data era, large-scale data processing systems are considered critical energy-consumption sources that exert significant impacts on the ecosystem. With the emergence of big data environments, which leads to the production of multi-structured data, numerous solutions and technologies have arisen to efficiently process vast amounts of data, to capitalize on real-time data generation. Recently, a centralized data repository platform called Data Lake has been proposed to manage the heterogeneous data originating from diverse sources, and sustainability issues in data platform management have gained traction. In this article, we intend to design and analyze an energy-aware strategy for a green data lake framework. This strategy prioritizes ecological concerns, particularly energy consumption, ensuring the maintenance of sufficient quality of service in terms of data availability. We present a model based on continuous-time Markov chain (CTMC) to analyze the trade-off between energy efficiency and performance of a green data lake platform.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Toward Green Data Lake Management and Analysis Through a CTMC Model

  • Marzieh Derakhshannia,
  • Julien Grange,
  • Nihal Pekergin

摘要

In the realm of big data era, large-scale data processing systems are considered critical energy-consumption sources that exert significant impacts on the ecosystem. With the emergence of big data environments, which leads to the production of multi-structured data, numerous solutions and technologies have arisen to efficiently process vast amounts of data, to capitalize on real-time data generation. Recently, a centralized data repository platform called Data Lake has been proposed to manage the heterogeneous data originating from diverse sources, and sustainability issues in data platform management have gained traction. In this article, we intend to design and analyze an energy-aware strategy for a green data lake framework. This strategy prioritizes ecological concerns, particularly energy consumption, ensuring the maintenance of sufficient quality of service in terms of data availability. We present a model based on continuous-time Markov chain (CTMC) to analyze the trade-off between energy efficiency and performance of a green data lake platform.