In this chapter, I present best practices for implementing complex data pipeline orchestration and observability in the AWS cloud. I discuss the criteria for choosing an orchestration solution and provide examples for orchestrating data pipelines with Amazon Managed Workflows for Apache Airflow (MWAA) and AWS Step Functions. I describe how to implement observability for data pipelines in the context of the three pillars of observability: logging, metrics, and traces. I’ll also demonstrate strategies for using generative AI to write our orchestration code and to help us observe and analyze these complex pipelines.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Data Pipeline Orchestration and Observability

  • Justin J. Leto

摘要

In this chapter, I present best practices for implementing complex data pipeline orchestration and observability in the AWS cloud. I discuss the criteria for choosing an orchestration solution and provide examples for orchestrating data pipelines with Amazon Managed Workflows for Apache Airflow (MWAA) and AWS Step Functions. I describe how to implement observability for data pipelines in the context of the three pillars of observability: logging, metrics, and traces. I’ll also demonstrate strategies for using generative AI to write our orchestration code and to help us observe and analyze these complex pipelines.