Data Engineering Part 1: Orchestrating Data Pipelines Using Databricks Workflows

The goal of orchestration is to configure multiple tasks into one complete end-to-end process or job. The orchestration service also needs to react to events or activities throughout the process and make decisions based on outputs from one automated task to determine and coordinate the next tasks. Finally, orchestration tools must provide full monitoring and observability capabilities to enable data engineers to have full visibility of their pipelines. See Figure 6-1.