Home Artificial Intelligence Monitoring Machine Learning Models in Production: Why and How?

Monitoring Machine Learning Models in Production: Why and How?

0
Monitoring Machine Learning Models in Production: Why and How?

How is our model impacted within the evolving world? An evaluation specializing in drift examples, and implementing Python-based monitoring strategies

Towards Data Science

Machine Learning (ML) model development often takes time and requires technical expertise. As data science enthusiasts, once we acquire a dataset to explore and analyze, we eagerly train and validate it using diverse state-of-the-art models or employing data-centric strategies. It feels incredibly fulfilling once we optimize the model’s performance as if all of the tasks have been completed.

Nonetheless, after deploying the model to production, there are many reasons that contribute to lower model performance or degradation.

Photo by Adrien Delforge on Unsplash

#1 The training data is generated through simulation

Data scientists often face limitations in accessing the production data, which leads to training the model using simulated or sample data as a substitute. While data engineers bear the responsibility of ensuring the representativeness of the training data by way of scale and complexity, the training data still deviates to some extent from the production data. There’s also a risk of systematic flaws in upstream data processing, akin to data collection and labeling. These aspects can impact the extraction of additional useful input features or hinder the model’s ability to generalize well.

Example: Investor data within the financial industry or patient information within the healthcare industry is usually simulated as a result of security and privacy concerns.

#2 The brand new production data exhibits a brand new data distribution

Over time, the characteristics of input features also can change, akin to shifts in age groups, income ranges, or other customer demographics. The info source itself may even be completely replaced as a result of various cases. Throughout the model development process, optimization relies on learning and capturing patterns from the bulk group throughout the training data. Nonetheless, as time progresses, the previous majority may transition into the minority within the production data, rendering the unique static model inadequate for meeting essentially the most recent production needs.

LEAVE A REPLY

Please enter your comment!
Please enter your name here