Apache Airflow is an open-source workflow management platform for data engineering pipelines. Airflow allow you to programmatically author and schedule workflows and monitor them via the built-in Airflow user interface. Airflow is written in Python, and workflows are created via Python scripts. Airflow is designed under the principle of “configuration as code”.

What about Airflow in the cloud? Cloud Composer is a managed version of Airflow that runs on Google Cloud Platform (GCP) and integrates well with other GCP services. Starting from November 2020, Amazon Web Services offers Managed Workflows for Apache Airflow.