Bayesian Optimization for Cloud Resource Management Through Machine Learning

Optimal cloud configuration of recurring big data analytic jobs is a relevant and challenging task in the industry. To this end, Bayesian Optimization is a promising method for efficiently finding optimal or near-optimal configurations for such applications, which are often executed in the cloud. On the other hand, Machine Learning methods can provide useful knowledge about the application at hand thanks to the quality of their estimations. In this paper, we propose a hybrid algorithm that is based on Bayesian Optimization and integrates elements from Machine Learning techniques to tackle time-constrained optimization problems in a cloud computing setting. We consider a recurring job scenario, where unfeasible points are to be avoided by all means, as they are a waste of resources. In such a context, Machine Learning helps to convey valuable information about the violation of constraints. Experiments on big data applications have shown that our algorithm significantly reduces the amount of unfeasible executions with respect to a pure constrained BO approach.