Home Artificial Intelligence Introduction to Data Version Control What’s Data Version Control (DVC)? Working with Hangar

Introduction to Data Version Control What’s Data Version Control (DVC)? Working with Hangar

0
Introduction to Data Version Control
What’s Data Version Control (DVC)?
Working with Hangar

PYTHON | DATA | PROGRAMMING

A step-by-step guide to implementing your personal DVC in Python using Hangar

Towards Data Science
Photo by Florian Olivo on Unsplash

Any production-level system requires some form of versioning.

A single source of current truth.

Any resources which can be constantly updated, especially concurrently by multiple users, require some form of an audit trail to maintain track of all changes.

In software engineering, the answer to that is Git.

If you will have written code in your life, you then are probably aware of the wonder that’s Git.

Git allows us to commit changes, create different branches from a source, and merge back our branches, to the unique to call a number of.

DVC is solely the identical paradigm but for datasets. See, live data systems are constantly ingesting newer data points while different users perform different experiments on the identical datasets.

This results in multiple versions of the identical dataset, which is unquestionably not a single source of truth.

Moreover, in a machine learning environment, we’d even have several versions of the identical ‘model’ trained on different versions of the identical dataset (for example, model re-training to incorporate newer data points).

If not properly audited and versioned, this is able to create a tangled web of datasets and experiments. We definitely don’t want that!

DVC is, due to this fact, a system that involves tracking our datasets by registering changes on a selected dataset. There are multiple DVC solutions each free and paid.

I recently discovered Hangar, a totally open-source Python DVC package. Let’s have a have a look at what it may do, lets?

The hangar package is a pure Python implementation and is on the market through pip.

Its core functionality can be closely developed to git, which greatly helps the training curve.

LEAVE A REPLY

Please enter your comment!
Please enter your name here