A free alternative to dbt cloud is Mage, an open-source data pipeline tool for data transformation and integration tasks.

Mage seamlessly complements dbt with a variety of advantages, including:

1. Integrated Web-based IDE: Mage provides a convenient web-based IDE where you possibly can develop and explore data models effortlessly inside a single interface.
2. Language Flexibility: With Mage, you possibly can mix the strengths of various tools and languages alongside dbt for enhanced data processing capabilities.
3. Visualizing dbt Model Output: Mage provides a built-in visualization capability, allowing users to effortlessly visualize the output generated by dbt models with just just a few clicks.
4. Data Extraction and Loading: Along with data transformation, Mage offers functionalities for data extraction and loading, enabling a more comprehensive end-to-end data pipeline solution.
5. Pipeline Scheduling and Retry Mechanism: Mage means that you can schedule your data pipelines and mechanically retry failed components, ensuring the sleek and reliable execution of your data integration processes.

Let’s dive deeper into each of those features.

Be at liberty to explore and experiment with the source code by cloning this GitHub repository:

Install Mage

You may install Mage using Docker, pip, or conda. This text will use Docker to put in Mage and initialize the project.

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]

For instance, let’s name our project “dbt_mage,” so the command becomes:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start dbt_mage

Find other ways to put in Mage here.

Create a pipeline

Open http://localhost:6789/ in your browser to view the Mage UI.

Click on “Latest” and choose “Standard (batch)” to create a brand new batch pipeline. Rename it as “dbt_pipeline.”

Install dependencies

Since we are going to use BigQuery as the info warehouse for dbt, we want to put in dbt-bigquery by adding it to the “requirements.txt” file and clicking on “Install packages.”

Create a dbt project

To create a dbt project, navigate to the suitable panel and click on on the terminal button.

Move to the “dbt” folder under your project and execute the command dbt init:

cd dbt_mage/dbt 
dbt init demo -s

This command adds the “demo” folder to the dbt directory.

Right-click on the “demo” folder and create a brand new file named “profiles.yml.” Specify your BigQuery credentials on this file.