Machine Learning Flow and Automated Pipelines

Putting a model into production is one of the most challenging tasks in the data science world. It is one of those last-mile problems that persists in many organizations. Although there are many tools for managing workflows, as the organization matures its needs change, and managing existing models can become a Herculean task. When you take a step back and analyze why it is so challenging, you can see that it is because of the structure that exists in most organizations. An engineering team maintains the production platform. There is a gap between the data science toolset and the production platforms. Some data science work can be developed in a Jupyter Notebook, with little consideration given to the cloud environment. Some of the data flows are created locally with limited scaling. Such applications tend to falter with large amounts of data. Best practices in the software development cycle don’t work well with the machine learning lifecycle because of the variety of tasks involved. The standard is mainly defined by the data science team in the organization. Also, rapid developments in the field are leaving vacuums with respect to the management and deployment of models.