The way in which you retrieve variables from Airflow can impact the performance of your DAGs

What happens if multiple data pipelines have to interact with the identical API endpoint? Would you actually must declare this endpoint in every pipeline? In case this endpoint changes within the near future, you should have to update its value in each file.

Airflow variables are easy yet helpful constructs, used to forestall redundant declarations across multiple DAGs. They’re simply objects consisting of a key and a JSON serializable value, stored in Airflow’s metadata database.

And what in case your code uses tokens or other variety of secrets? Hardcoding them in plain-text doesn’t appear to be a secure approach. Beyond reducing repetition, Airflow variables also aid in managing sensitive information. With six alternative ways to define variables in Airflow, choosing the suitable method is crucial for ensuring security and portability.

An often missed aspect is the impact that variable retrieval has on Airflow performance. It could possibly potentially strain the metadata database with requests, each time the Scheduler parses the DAG files (defaults to thirty seconds).

It’s fairly easy to fall into this trap, unless you understand how the Scheduler parses DAGs and the way Variables are retrieved from the database.

Before moving into the discussion of how Variables are fetched from the metastore and what best practices to use in an effort to optimise DAGs , it’s essential to get the fundamentals right. For now, let’s just give attention to how we will actually declare variables in Airflow.

As mentioned already, there are several alternative ways to declare variables in Airflow. A few of them become safer and portable than others, so let’s examine all and take a look at to know their pros and cons.

1. Making a variable from the User Interface

In this primary approach, we’re going to create a variable through the User Interface. From the highest menu select Admin → Variables → +