CLS-Luigi: Analytics Pipeline Synthesis
摘要
We present CLS-Luigi, a framework for synthesizing analytics pipelines that enable prediction and decision-making. Analytics pipelines typically consist of a number of diverse steps ranging from simple preprocessing to informed machine learning and optimization algorithms. Implementing pipelines and selecting algorithms is time-consuming and the performance of selected algorithms is interdependent. CLS-Luigi improves the implementation process by adapting and using a well-established synthesis framework, Combinatory Logic Synthesizer (CLS), to automatically generate pipeline variants based on a repository of typed components that are implemented in Python. Luigi, a pipeline framework developed by Spotify, executes pipeline variants efficiently and allows intermediate results to be shared among similar pipelines to optimize resource utilization. We demonstrate the simplicity, expressive power, and run-time gains of CLS-Luigi through two examples: First, we show that CLS-Luigi has the modeling capabilities to automatically generate machine learning pipelines that cover all algorithms included in AutoSklearn, and delivers high-quality results. Second, we go beyond the capabilities of AutoML by facilitating the consistent synthesis and execution of decision pipelines covering different optimization paradigms. Our framework is easily accessible, open-source, and compatible with most existing tools and Python libraries used in data analytics.