Orchestrating Data Engineering Pipelines using Prefect

Earlier we looked at Apache Airflow, the most widely adopted workflow orchestration solution. Apache Airflow is well adopted and supported by the community. In this chapter, we will look at another workflow orchestration solution called Prefect. We will look at its architecture and how you can create a workflow without DAGs. You can still define DAGs in Prefect though. The features, components, and typical workflow of Prefect appear different from what we have seen with Apache Airflow. As for which tool is a better choice, it depends on many factors like team, requirements, and infrastructure, among other things.