Apache Airflow in Nutshell

What is Airflow?

Airflow is a platform to Programmatically Author, Schedule and Monitor Workflows or Data Pipeline.

What is Workflow?

§ A Sequence of tasks.

§ Schedule or Triggered an email

§ Frequently used to handle big data processing pipeline.

E.g. of Workflow

Traditional ETL Approach:

Time Based Scheduler in UNIX like Computers for Daily Extraction is required if Data is generating every day.

Source-> CRON Script-> DB

These CRON Scripts has lot of problem and not scalable hence need of Airflow comes into the picture.

Some Problems with CRON jobs:

1) Failures-> if Failure happens. (How many time? How often?)

2) Monitoring -> Success or Failure States.

3) Dependencies -> Execution Dependencies.

4) Scalability -> New Scheduler is hard to add.

5) Deployment -> Deploy New Changes.

6) Process historic data -> Rerun Historic Data.

Many Problem one Solution -> Apache Airflow

Introduction

Airflow Framework to define tasks and dependencies in python, Executing, Scheduling, Distributing tasks across worker Nodes. View of present and past runs, logging feature Extensible through plugins. Nice UI, Possibility to define REST interface, Interact well with databases.

Airflow DAG (Directed Acyclic Graph) with Multiple tasks which can be executed independently or interdependently.

Benefits:

1) Can handle upstream | downstream dependencies.

2) Jobs can pass parameters to other jobs.

3) Handle Errors and failures.

4) Data Sensors to trigger a DAG when data arrives.

5) Job testing through airflow itself.

6) Monitoring and email alerts.

7) Community supports

8) Accessibility of log files.

9) Implementing Trigger rules.

So as you get Familiar with Apache Airflow and its benefits. Let’s write our first Program in Airflow-Python.

Writing Your First Airflow Program consists of below steps:

Step1: Importing Modules.

Step2: Default Arguments.

Step3: Instantiate a DAG.

Step4: Tasks.

Step5: Setting up Dependencies.

Prerequisite:

Install Python and Airflow setup on your Machine.

Refer this: https://medium.com/@jacksonbull1987/how-to-install-apache-airflow-6b8a2ae60050

Step 1: Importing Modules

Import Python dependencies

Airflow schedules the job on the principal of Directed Acyclic Graph i.e. DAG.

Hence we need to import DAG. If we wrote any Python Function or Shell Script according to that we have to import Operators.

We can also import date time or in built date function of airflow.

There are various modules and functionality airflow offers so do explore further as per of your requirements.

Step 2: Default Arguments

Here, we all define some default arguments. Which can be like Owner, Dag should depend on past jobs or not, whom to mail if failure occurs.

If failure occur how many times it should retries to execute the dag in how much interval of time.

Step 3: Instantiate a DAG.

Here we are Instantiate DAG just like objects in case of classes.

We need to pass all the arguments to constructor so that it didn’t give any errors.

Schedule Interval is used for how often the tasks should get executed.

Find Below chart for better understanding.

Step 4: Tasks.

Here we can define any number of python function (or Bash Functions, etc.). Which will act as Tasks. We can write any ETL jobs inside these functions and execute as per convenience.

Step 5: Setting up Dependencies.

In which order task should be executed in.

t1.set_downstream(t2) == t1>>t2 (This means that t2 will depend on t1)

t1.set_upstream(t2) == t1<<t2 (This means that t1 will depend on t2)

for multiple task dependencies.

t1>>[t2,t3] (This means that t1 will depend on t2 and t3)

[t2,t3]<<t1 (This means that t2,t3 will depend on t1)

Follow for more Technical stories!!

--

--

--

Lead Data Scientist at Tata Power Ltd.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Dev Day 50: Allowing Shields To Take 3 Hits Unity3D

Design RESTful API using Swagger OpenAPI Specification 3.0

Api versioning in Asp.Net Core

Invoking HTTP Requests from Selenium using HttpClient

Update on 2022 Grants and Compensation Schedule

Beyond A/B Testing: How to Optimize Anything

How to use root() and vpa() from symbolic math toolbox in simulink?

If your RPA vendor says this… Walk Away

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vikas Maurya

Vikas Maurya

Lead Data Scientist at Tata Power Ltd.

More from Medium

Creating a Randomly Sampled Working Data in Spark SQL and Python from Original Dataset

A slice of a pie

All In One Custom PostgreSQL For Database Developer Docker Image

ETL jobs with retry option using Kafka

Data pipeline automation using Airflow DAGs in Docker containers