Data Pipelines with Apache Airflow, Second Edition

Data Pipelines with Apache Airflow, Second Edition is a guide to building, monitoring and maintaining data workflows with Apache Airflow. This edition has been fully revised for Airflow 3 and covers its newer features, including the Taskflow API, deferrable operators, event-driven scheduling, dynamic task mapping, DAG versioning, the redesigned UI and integration with large language models. It takes readers from beginner to expert through real-world scenarios and hands-on examples, and works as both a tutorial and a reference.

The book explains the core ideas behind Airflow's architecture and workflow design, then moves on to practical skills. Readers learn to deploy pipelines as Airflow DAGs, schedule them with the Dataset API and timetables (including complex irregular schedules), and use both time-based and event-based strategies. They also learn to write custom Airflow components, test their pipelines thoroughly, and follow industry best practices for building and maintaining workflows.

Other topics include running Airflow in production, deploying it with Kubernetes, and orchestrating workflows in container-native environments. The examples cover common tasks such as ingesting and transforming data and connecting to several data sources. They also include AI-oriented work such as building RAG systems and training and deploying machine learning and generative AI models, with projects ranging from model training to optimizing delivery routes.

Airflow itself is a platform for collecting, consolidating, cleaning and analyzing data, with a friendly UI, scheduling and monitoring features, and flexible Python scripting. The book is aimed at data engineers, machine learning engineers, DevOps engineers and sysadmins with intermediate Python skills. It is written by Julian de Ruiter, Ismael Cabral, Kris Geusebroek, Daniel van der Ende and Bas Harenslak, all experienced data engineers and Airflow experts.