We learned where we are actually and where we’re going with AutoML. The query is how we’re getting there. We summarize the issues we face today into three categories. When these problems are solved, AutoML will reach mass adoption.

Problem 1: Lack of business incentives

Modeling is trivial compared with developing a usable machine learning solution, which can include but is just not limited to data collection, cleansing, verification, model deployment, and monitoring. For any company that may afford to rent people to do all these steps, the price overhead of hiring machine learning experts to do the modeling is trivial. After they can construct a team of experts without much cost overhead, they don’t trouble experimenting with recent techniques like AutoML.

So, people would only start to make use of AutoML when the prices of all other steps are reduced to the underside. That’s when the price of hiring people for modeling becomes significant. Now, let’s see our roadmap towards this.

Many steps might be automated. We ought to be optimistic that because the cloud services evolve, many steps in developing a machine learning solution may very well be automated, like data verification, monitoring, and serving. Nevertheless, there may be one crucial step that may never be automated, which is data labeling. Unless machines can teach themselves, humans will all the time need to arrange the information for machines to learn.

Data labeling may develop into the important cost of developing an ML solution at the top of the day. If we will reduce the price of knowledge labeling, they might have the business incentive to make use of AutoML to remove the modeling cost, which could be the one cost of developing an ML solution.

The long-term solution: Unfortunately, the final word solution to scale back the price of knowledge labeling doesn’t exist today. We’ll depend on future research breakthroughs on “learning with small data”. One possible path is to take a position in transfer learning.

Nevertheless, people should not all for working on transfer learning since it is tough to publish on this topic. For more details, you’ll be able to watch this video, Why most machine learning research is useless.

The short-term solution: Within the short-term, we will just fine-tune the pretrained large models with small data, which is an easy way of transfer learning and learning with small data.

In summary, with a lot of the steps in developing an ML solution automated by cloud services, and AutoML can use pretrained models to learn from smaller datasets to scale back the information labeling cost, there will likely be business incentives to use AutoML to scale back their cost in ML modeling.

Problem 2: Lack of maintainability

All deep learning models should not reliable. The behavior of the model is unpredictable sometimes. It is tough to know why the model gives specific outputs.

Engineers maintain the models. Today, we’d like an engineer to diagnose and fix the model when problems occur. The corporate communicates with the engineers for anything they need to vary for the deep learning model.

The AutoML system is far harder to interact with than an engineer. Today, you’ll be able to only use it as a one-shot method to create the deep learning model by giving the AutoML system a series of objectives clearly defined in math prematurely. Should you encounter any problem using the model in practice, it should not assist you to fix it.

The long-term solution: We want more research in HCI (Human-Computer Interaction). We want a more intuitive method to define the objectives in order that the models created by AutoML are more reliable. We also need higher ways to interact with the AutoML system to update the model to satisfy recent requirements or fix any problems without spending an excessive amount of resources searching all the several models again.

The short-term solution: Support more objective types, like FLOPS and the variety of parameters to limit the model size and inferencing time, and weighted confusion matrix to cope with imbalanced data. When an issue occurs within the model, people can add a relevant objective to the AutoML system to let it generate a brand new model.

Problem 3: Lack of infrastructure support

When developing an AutoML system, we found some features we’d like from the deep learning frameworks that just don’t exist today. Without these features, the facility of the AutoML system is proscribed. They’re summarized as follows.

First, state-of-the-art models with flexible unified APIs. To construct an efficient AutoML system, we’d like a big pool of state-of-the-art models to assemble the ultimate solution. The model pool must be updated commonly and well-maintained. Furthermore, the APIs to call the models must be highly flexible and unified so we will call them programmatically from the AutoML system. They’re used as constructing blocks to construct an end-to-end ML solution.

To resolve this problem, we developed KerasCV and KerasNLP, domain-specific libraries for computer vision and natural language processing tasks built upon Keras. They wrap the state-of-the-art models into easy, clean, yet flexible APIs, which meet the necessities of an AutoML system.

Second, automatic hardware placement of the models. The AutoML system may have to construct and train large models distributed across multiple GPUs on multiple machines. An AutoML system ought to be runnable on any given amount of computing resources, which requires it to dynamically determine distribute the model (model parallelism) or the training data (data parallelism) for the given hardware.

Surprisingly and unfortunately, not one of the deep learning frameworks today can mechanically distribute a model on multiple GPUs. You’ll have to explicitly specify the GPU allocation for every tensor. When the hardware environment changes, for instance, the variety of GPUs is reduced, your model code may not work.

I don’t see a transparent solution for this problem yet. We must allow a while for the deep learning frameworks to evolve. Some day, the model definition code will likely be independent from the code for tensor hardware placement.

Third, the benefit of deployment of the models. Any model produced by the AutoML system may must be deployed down the stream to the cloud services, end devices, etc. Suppose you continue to have to hire an engineer to reimplement the model for specific hardware before deployment, which is probably the case today. Why don’t you only use the identical engineer to implement the model in the primary place as a substitute of using an AutoML system?

Individuals are working on this deployment problem today. For instance, Modular created a unified format for all models and integrated all the key hardware providers and deep learning frameworks into this representation. When a model is implemented with a deep learning framework, it might be exported to this format and develop into deployable to the hardware supporting it.

With all the issues we discussed, I’m still confident in AutoML in the long term. I imagine they will likely be solved eventually because automation and efficiency are the long run of deep learning development. Though AutoML has not been massively adopted today, it should be so long as the ML revolution continues.