Big data analytics, powered by tools like Apache Spark, is revolutionizing organizational decision-making by transforming vast datasets into actionable insights, optimizing processes, and enhancing competitiveness across industries. However, in shared Spark clusters, inefficient resource allocation due to inconsistent expertise among data scientists often leads to suboptimal performance. This paper addresses the challenge of defining effective hardware resource configurations for Spark-based analytics tasks within an organization. We introduce the Subset Selection Problem for Hardware Resource Configuration, a combinatorial optimization problem aimed at minimizing total computational time by selecting a limited set of predefined configurations from historical job data. Two solution approaches are proposed: a branch-and-bound algorithm and a branch-and-cut algorithm based on a novel mixed integer programming formulation. Extensive experiments on real-world simulated instances demonstrate that the MIP approach outperforms the branch-and-bound method in both scalability and efficiency, offering a practical solution for optimizing resource utilization in Spark clusters.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Mixed Integer Program for Selection of Hardware Resource Configurations in a Spark Cluster

  • Quoc-Trung Bui,
  • Nguyen Van-Trong,
  • Huynh Thi-Thanh-Binh

摘要

Big data analytics, powered by tools like Apache Spark, is revolutionizing organizational decision-making by transforming vast datasets into actionable insights, optimizing processes, and enhancing competitiveness across industries. However, in shared Spark clusters, inefficient resource allocation due to inconsistent expertise among data scientists often leads to suboptimal performance. This paper addresses the challenge of defining effective hardware resource configurations for Spark-based analytics tasks within an organization. We introduce the Subset Selection Problem for Hardware Resource Configuration, a combinatorial optimization problem aimed at minimizing total computational time by selecting a limited set of predefined configurations from historical job data. Two solution approaches are proposed: a branch-and-bound algorithm and a branch-and-cut algorithm based on a novel mixed integer programming formulation. Extensive experiments on real-world simulated instances demonstrate that the MIP approach outperforms the branch-and-bound method in both scalability and efficiency, offering a practical solution for optimizing resource utilization in Spark clusters.