<p>With the rapid development of big data and real-time stream processing technologies, Apache Flink, as one of the most mainstream stream processing frameworks today, is widely used for large-scale data processing and real-time analysis. However, in resource-level heterogeneous cluster environments, the default data partitioning strategy of Flink suffers from load imbalance and low resource utilization. Existing studies primarily focus on load balancing between cluster nodes while neglecting the uneven distribution of loads among parallel task instances. To address these issues, this paper constructs a three-tier load balancing model, which considers the load balancing optimisation of clusters comprehensively from the three levels of task instances, computing nodes and data transmission. Based on this model, load-aware dynamic data partitioning algorithm (LADP) is proposed, which is designed to not only achieve load balancing between cluster nodes and task instances during partitioning but also prioritize low-latency downstream nodes to alleviate load skew among task instances and reduce data transmission latency. Additionally, through load skew repair strategy and dynamic balanced allocation strategy, LADP can adaptively adjust data distribution for overloaded nodes, making stream data distribution more balanced. Experimental results show that compared to the default partitioning strategy, Dr-Stream, DPS, and St-Stream, LADP increases the system’s average throughput by 17.83%, 2.40%, 21.14%, and 7.76%, respectively, and reduces average latency by 12.91% and 8.41%, 20.20% and 4.24%. The study in this paper is of great significance for improving the performance of Flink in resource-level heterogeneous cluster environments, and provides a more efficient solution for large-scale data processing and real-time analysis.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Three-tier Load Balancing Model with Dynamic Data Partitioning Strategy for Distributed Stream Processing

  • Hongjian Li,
  • Yifan Ren,
  • Wei Luo,
  • Xiaolin Duan

摘要

With the rapid development of big data and real-time stream processing technologies, Apache Flink, as one of the most mainstream stream processing frameworks today, is widely used for large-scale data processing and real-time analysis. However, in resource-level heterogeneous cluster environments, the default data partitioning strategy of Flink suffers from load imbalance and low resource utilization. Existing studies primarily focus on load balancing between cluster nodes while neglecting the uneven distribution of loads among parallel task instances. To address these issues, this paper constructs a three-tier load balancing model, which considers the load balancing optimisation of clusters comprehensively from the three levels of task instances, computing nodes and data transmission. Based on this model, load-aware dynamic data partitioning algorithm (LADP) is proposed, which is designed to not only achieve load balancing between cluster nodes and task instances during partitioning but also prioritize low-latency downstream nodes to alleviate load skew among task instances and reduce data transmission latency. Additionally, through load skew repair strategy and dynamic balanced allocation strategy, LADP can adaptively adjust data distribution for overloaded nodes, making stream data distribution more balanced. Experimental results show that compared to the default partitioning strategy, Dr-Stream, DPS, and St-Stream, LADP increases the system’s average throughput by 17.83%, 2.40%, 21.14%, and 7.76%, respectively, and reduces average latency by 12.91% and 8.41%, 20.20% and 4.24%. The study in this paper is of great significance for improving the performance of Flink in resource-level heterogeneous cluster environments, and provides a more efficient solution for large-scale data processing and real-time analysis.