Analysis of Job Processing Data – Towards Large Cloud Infrastructure Operation Simulation

Cloud computing is the most popular way of delivering on-demand computational resources. Recently, the research in this area has started to focus on carbon-aware clouds. Here, the most challenging aspects are related to defining strategies for efficient task scheduling and resource allocation. These strategies can be simulated and assessed using dedicated tools. However, to perform their accurate evaluation, the tests should reproduce close-to-real conditions of the actual cloud center. In particular, they require running simulations with various mixtures of tasks that replicate the actual cloud center operation. Therefore, the main aim of this work was to prepare tools that will allow the generation of synthetic job streams, with mixes of realistic types of computational tasks. The core of this contribution is the analysis of actual job processing data from the CloudFerro cloud center. The proposed methodology is based on data clustering and includes a comparison between multiple algorithms. Furthermore, the resulting clusters have been categorized from the point of view of cloud center operation, in order to identify prototypical tasks’ classes with respect to the resource demands. Finally, a tool that generates synthetic job streams, based on the Gaussian Mixture Model, which has been implemented, is summarized.