Apache Airflow Bootcamp: Hands-On Workflow Automation

Step-by-Step Guide to Building and Managing Robust Workflows with Apache Airflow

Created by: Sriw World of Coding
Instructor at Udemy

What Will I Learn?

• Understand what Apache Airflow is, its purpose and pros and cons of using Airflow
• Step-by-step guide to installing Airflow
• Launch and navigate the Airflow Web UI and learn about various views: DAG, Grid, Graph, Calendar, Task Duration, Code, Variable and Gantt View
• Understand what a DAG is and how to create a DAG definition file and different methods for DAG creation
• Learn about DAG Run, default_arguments, and DAG arguments and Master scheduling concepts such as depends_on_past, wait_for_downstream, catchup, and backfill
• Use the Airflow CLI for various operations and access a handy cheatsheet for quick reference
• Understand tasks, task instances and Learn the lifecycle of a task
• Master different operators including BashOperator, PostgresOperator, PythonOperator, SqliteOperator, and EmailOperator
• Implement sensors like FileSensor, SQLSensor, TimeDeltaSensor, and TimeSensor
• Apply branching logic with BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, and ShortCircuitOperator
• Manage DAG dependencies and use TaskGroups ,Utilize TriggerDagRunOperator , ExternalTaskSensor and use hooks such as PostgresHook and SHook
• Manage resources with pools and task priorities
• Learn about different types of executors: SequentialExecutor and LocalExecutor and learn the Transition from SequentialExecutor to LocalExecutor
• Explore the Airflow metadata database and Manage roles and create users with different roles including admin, public, user, and operator roles
• Set and manage task-level and DAG-level SLAs and handle SLA misses
• Address issues like zombie tasks, SIGTERM, and SIGKILL errors

Requirements

• Knowledge of Python
• Rest we will cover to learn Airflow from scratch
• Familiarity with command-line interfaces.
• Understanding of database concepts is a plus but not required

Target audience

• Data Engineers: Data engineers who are responsible for building and managing data pipelines can greatly benefit from learning Apache Airflow.
• Data Scientists: Data scientists who work with large datasets and perform data analysis can leverage Apache Airflow to automate repetitive tasks, such as data preprocessing, model training, and evaluation
• DevOps Engineers: DevOps engineers who are responsible for managing and automating infrastructure can use Apache Airflow to automate deployment processes, monitor system health, and trigger actions based on predefined conditions
• Software Developers: Software developers who build and maintain software applications can use Apache Airflow to automate various tasks, such as data ingestion, data processing, and workflow orchestration