Stream Economics: Resource Efficiency in Streams with Task Over-Allocation and Load Shedding

In this paper we propose an alternative task scheduling mechanism for stream processing systems such as Apache Flink, that targets resource efficiency in a multi-tenant stream processing environment with several resource heterogeneous tasks being executed in parallel. The task scheduler we propose doesn’t limit the amount of tasks that can run on each machine, instead, it adapts tasks’ allocation based on their runtime metrics, scheduling tasks to the machines with more available resources. At the same time, we explore load shedding in stream processing applications, as a mechanism to solve the tasks’ resource starvation problem that may appear due to bad decisions performed by the scheduler, because of its optimistic approach and due to the dynamic workloads of the applications. We implemented a proof-of-concept of such system in Apache Flink and tested it against scenarios that show the different aspects and advantages of the developed mechanism in action.