Scheduling ensemble workflows on hybrid resources in IaaS clouds

Scientific ensemble workflows are commonly executed in Infrastructure-as-a-Service clouds for high-performance computing. The dynamic pricing of spot instances offers a cost-effective way for users to rent cloud resources. However, these instances are subject to out-of-bid failures when their prices exceed the user’s bid, leading to task termination and disruptions in workflow execution. It is a great challenge to reduce costs while ensuring the quality of task completion. This paper addresses the problem of scheduling prioritized ensemble workflows using on-demand and spot instances, with the objective of maximizing the number of high-priority workflows completed while minimizing total cost. We propose a rules-based scheduling heuristic with hybrid provisioning, which includes task scheduling, dynamic provisioning, and spot monitoring processes. The proposed algorithm is evaluated by comparing it to existing algorithms for similar problems over two classic scientific workflow datasets, Montage and LIGO. The score for completing as many high-priority workflows as possible is calculated within the given deadline D. The results reveal that our proposed algorithm achieves an average 30% improvement in the RPD value at different deadline levels and task sizes than other baseline algorithms.