Scholarlink

<p>With the rapidly growing demand for real-time processing of large amounts of stream data, the global deployment of cloud infrastructure for stream processing has brought tremendous economic benefits to cloud service providers, yet large-scale stream processing continues to increase resource cost consumption. In stream processing, the trade-off between performance and cost for workflows scheduling at lower resource cost remains to be explored. In this paper, a deployment method of streaming applications is proposed to maximize the performance-to-cost ratio in Storm, and a communication traffic optimization method for inter-node communication based on graph partitioning is proposed to reduce resource cost. First, we propose a cost-efficient model based on performance-to-cost ratio and a Storm-based performance-to-cost ratio maximization deployment algorithm PC-Storm. The PC-Storm first detects and collects task node information in the heterogeneous cluster. Then, it calculates the performance-to-cost ratio of the nodes and builds a performance-to-cost ratio table. Finally, the Storm task threads are assigned to nodes with the high performance-to-cost ratio. In addition, a maxSubtopology algorithm is also proposed to optimize the communication traffic between Storm nodes based on graph partitioning. The maxSubtopology algorithm assigns the tasks with heavy communication traffic to the same node under the premise of satisfying the performance constraint. Compared with the existing algorithms, the PC-Storm improves the resource performance-to-cost ratio by 40.92%, and the maxSubtopology algorithm reduces the communication cost by 42.8%. The proposed method can maximize performance-to-cost ratio and reduce the communication cost effectively for deployment of streaming applications.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adaptive scheduling framework of streaming applications based on performance-to-cost ratio in heterogeneous cloud environment

  • 期刊论文

  • 发表日期：

    2025年4月11日

查看全文
  • Hongjian Li,
  • Gangfan Tan,
  • Chenzi Wang,
  • Yuan Gao,
  • Tao Zhou,
  • Xiaolin Duan

摘要

With the rapidly growing demand for real-time processing of large amounts of stream data, the global deployment of cloud infrastructure for stream processing has brought tremendous economic benefits to cloud service providers, yet large-scale stream processing continues to increase resource cost consumption. In stream processing, the trade-off between performance and cost for workflows scheduling at lower resource cost remains to be explored. In this paper, a deployment method of streaming applications is proposed to maximize the performance-to-cost ratio in Storm, and a communication traffic optimization method for inter-node communication based on graph partitioning is proposed to reduce resource cost. First, we propose a cost-efficient model based on performance-to-cost ratio and a Storm-based performance-to-cost ratio maximization deployment algorithm PC-Storm. The PC-Storm first detects and collects task node information in the heterogeneous cluster. Then, it calculates the performance-to-cost ratio of the nodes and builds a performance-to-cost ratio table. Finally, the Storm task threads are assigned to nodes with the high performance-to-cost ratio. In addition, a maxSubtopology algorithm is also proposed to optimize the communication traffic between Storm nodes based on graph partitioning. The maxSubtopology algorithm assigns the tasks with heavy communication traffic to the same node under the premise of satisfying the performance constraint. Compared with the existing algorithms, the PC-Storm improves the resource performance-to-cost ratio by 40.92%, and the maxSubtopology algorithm reduces the communication cost by 42.8%. The proposed method can maximize performance-to-cost ratio and reduce the communication cost effectively for deployment of streaming applications.