DATA ENGINEERING

A go-to guide for data engineers wading through the backfilling maze

Imagine starting a brand new data pipeline and getting data from a source you’ve never parsed before (e.g. pulling info from an API or an existing hive table). Now, you’re on a mission to make it look like you collected this data ages ago. That’s one example of what we call data backfilling in data engineering.

Nevertheless it’s not nearly starting a brand new data pipeline or table. You could possibly have a table that’s been gathering data for some time, and suddenly, it’s worthwhile to change the information (for instance resulting from a brand new metric definition), or toss in additional data from a brand new data source. Or perhaps there’s a clumsy gap in your data, and you only need to patch it up. All these situations are examples of information backfilling. The common thread is popping “back” in time and “filling” up your table with some historical data.

The next figure (Figure 1) shows an easy backfilling scenario. On this instance, a each day job retrieves data from two upstream sources (one for platform A and one other for platform B). The dataset is structured with the primary partition being ‘ds,’ and the second partition (or sub-partitions) representing the platforms. Unfortunately, data for the period from 2023–10–03 to 2023–10–05 is absent resulting from certain issues. To deal with this gap, a backfilling operation was initiated (the backfilling job began on 2023–10–08).

A temporary heads-up before proceeding further: inside the domain of information engineering, we normally encounter two scenarios: “backfilling” a table or “restating” a table. These processes, while sharing some similarities, have some subtle differences. Backfilling, as a practice, is about populating missing or incomplete data in a dataset. Its application is often directed towards updating historical data or rectifying gaps. Conversely, restating a table involves effecting substantial…