Strategies for Overcoming the Challenges of Scaling Open-Source AI Models in Production
For those who’re reading this text, you almost certainly need no introduction to some great benefits of deploying open-source models. Over the past couple of years, we’ve seen incredible growth within the each the amount and quality of open source models.
- Platforms resembling Hugging Face have democratized access to a big selection of models, including Large Language Models (LLMs) and diffusion models, empowering developers to innovate freely and efficiently.
- Developers enjoy greater autonomy, as they will fine-tune and mix different models at will, resulting in modern approaches like Retrieval-Augmented Generation (RAG) and the creation of advanced agents.
- From an economic perspective, open-source models provide substantial cost savings, enabling using smaller, specialized models which might be more budget-friendly in comparison with general-purpose models like GPT-4.
Open-source models present a pretty solution, but what’s the following hurdle? Unlike using a model endpoint like OpenAI, where the model is a scalable black box behind the API, deploying your individual open-source models introduces scaling challenges. It’s crucial to be certain that your model scales effectively with production traffic and maintains a seamless experience during traffic spikes. Moreover, it’s necessary to administer costs efficiently, so you simply pay for what you utilize and avoid any financial surprises at the tip of the month.
Interestingly, this appears like a challenge that modern serverless architectures, like AWS Lambda, have already solved — an answer which have existed for nearly a decade. Nonetheless, in terms of AI model deployment, this isn’t quite the case.
The restrictions of serverless functions for AI deployments are multifaceted.
- No GPU support. Platforms like AWS Lambda don’t support GPU. This isn’t merely a technical oversight; it’s rooted in architectural and practical considerations.
- GPUs can’t be easily shared. GPUs, while highly parallelizable as devices, isn’t as flexible in handling multiple inference tasks on different models concurrently.
- GPUs are expensive. They’re exceptional for model inferencetasks but costly to keep up, especially if not utilized repeatedly.
Next, let’s take a take a look at our scaling journey and the necessary lessons we’ve learned along the way in which.
Before we could even begin to work on scaling, we’ve the notorious “cold start” problem. This issue presents itself in three different stages:
- Cloud provisioning: This phase involves the time it takes for a cloud provider to allocate an instance and integrate it into our cluster. This process varies widely, starting from as quick as 30 seconds to several minutes, and in some cases, even hours, especially for high-demand instances just like the Nvidia A100 and H100 GPUs.
- Container image pulling: Unlike easy Python job images, AI model serving images are very complex, because of the dependencies and custom libraries they require. Although cloud providers boast multi-gigabit network bandwidth, our experience often saw download speeds far below them, with image pulling time about 3 minutes.
- Model loading. The time required here is basically depending on the model’s size, with larger models like LLMs and diffusion models taking significantly longer time because of their billions of parameters. For instance, loading a 5GB model like Stable Diffusion 2 might take roughly 1.3 minutes with 1Gbps network bandwidth, while larger models like Llama 13B and Mixtral 8x7B could require 3.5 minutes and 12.5 minutes respectively.
Each phase of the cold start issue demands specific strategies to reduce delays. In the next sections, we’ll explore each of them in additional detail, sharing our strategies and solutions.
Cloud provisioning
In contrast to the homogeneous environment of serverless CPUs, managing a various range of compute instance types is crucial when coping with GPUs, each tailored for specific use cases. As an example, IO-bound LLMs require high GPU memory bandwidth and capability, while generative models need more powerful GPU compute.
Ensuring availability during peak traffic by maintaining all GPU instance types may lead to prohibitively high costs. To avoid the financial strain of idle instances, we implemented a “standby instances” mechanism. Slightly than preparing for the utmost potential load, we maintained a calculated variety of standby instances that match the incremental scaling step sizes. For instance, if we scale by two GPUs at a time, we want to have two standby instances ready. This permits us to quickly add resources to our serving fleet as demand surges, significantly reducing wait time, while keeping cost manageable.
In a multi-tenant environment, where multiple teams or, in our case, multiple organizations, share a typical resource pool, we will achieve more efficient utilization rates. This shared environment allows us to balance various resource demands, contributing to improved cost efficiency. Nonetheless, managing multi-tenancy introduces challenges, resembling enforcing quotas and ensuring network isolation, which might add complexity to the cluster.
Container image pulling
Serverless CPU workloads often use lightweight images, just like the Python slim image (around 154 MB). In stark contrast, a container image built for serving an LLM could be much larger (6.7 GB); the majority of this size comes from the assorted dependencies required to run the AI model.
Despite high-bandwidth networks advertised by cloud providers, the truth often falls short, with actual download speeds being a fraction of the promised rates.
Practically, a good portion of the files were never used. A technique is to optimize the container image itself, but that quickly proved to be unmanageable. As a substitute, we shifted our focus to an on-demand file pulling approach. Specifically, we first downloaded only the image metadata, with the actual distant files being fetched later as needed. As well as, we leveraged peer-to-peer networking throughout the cluster to dramatically increase pulling efficiency.
With these optimizations, we reduced the image pulling time from several minutes to mere seconds. Nonetheless, everyone knows this measurement is “cheating” for the reason that actual files should not pulled at this stage. The actual file pulling occurs when the service runs. Subsequently, it’s crucial to have a service framework that lets you define behaviors at various lifecycle stages, resembling initialization and serving. By doing the entire bootstrapping during initialization, we will be certain that all file dependencies are pulled. This manner, in terms of serving time, there aren’t any delays attributable to file pulling.
Within the above example, model loading is completed in the course of the initialization lifecycle inside __init__ and serving happens throughout the @bentoml.api named txt2img.
Model loading
Initially, probably the most straightforward method for model loading was to fetch it directly from a distant store like Hugging Face. Using Content Delivery Networks (CDNs), NVMe SSDs, and shared memory, we could remove among the bottlenecks. While this worked, it was removed from optimal.
To enhance this process, we considered using in-region network bandwidth. We seeded models in our distributed file systems and broke them into smaller chunks, allowing for parallel downloads. This drastically improved performance, but we still encountered cloud provider’s network bandwidth bottlenecks.
In response, we further optimized to leverage in-cluster network bandwidth by utilizing peer-to-peer sharing and tapping into local caches. While the improvements were substantial, they added a layer of complexity to the method, which we want to abstract away from the developers.
Even with the above practices, we still suffer from a sequential bottleneck: the necessity to wait for every step to finish before proceeding with the following. Models needed to be downloaded to persistent drive entirely before loading into CPU memory, after which into the GPU.
We turned to a stream-based method for loading model weights, using the distributed file cache system we had in place. This technique allows programs to operate as if all files were logically available on disk. In point of fact, the required data is fetched on-demand from distant storage due to this fact bypassed disk writing. By leveraging a format like Safetensors, we will efficiently load the model weights into the major memory through memory mapping (mmap) before loading to the GPU memory in a streaming fashion.
Furthermore, we adopted asynchronous writing to disk. By doing so, we created a faster-access cache layer on the local disk. Thus, latest deployments with only code changes could bypass the slower distant storage fetch phase, reading the model weights from local cache directly.
To summarize, we managed to optimize the cold start time and we were blissful with the outcomes:
- No cloud provision delay with standby instances.
- Faster container image pulling with on-demand and peer-to-peer streaming.
- Accelerated model loading time with distributed file systems, peer-to-peer caching, and streamed loading to GPU memory.
- Parallelized image pulling and model loading enabled by service framework.
Next, we want to discover probably the most indicative signal for scaling AI model deployments on GPUs.
Resource utilization metrics
Initially, we considered CPU utilization. It’s straightforward and has an intuitive default threshold, resembling 80%. Nonetheless, the plain drawback is that CPU metrics don’t capture GPU utilization. Moreover, the Global Interpreter Lock (GIL) in Python limits parallelism, stopping high CPU utilization on multi-core instances, making CPU utilization a less feasible metric.
We also explored GPU utilization as a more direct measure of our models’ workloads. Nonetheless, we encountered a problem: the GPU utilization reported by tools like nvml didn’t accurately represent the actual utilization of the GPU. This metric samples kernel usage over a time period, and a GPU is taken into account utilized if at the very least one kernel is executing. This aligns with our statement that higher performance can often be achieved through improved batching, though the GPU device was already reported as having high utilization.
Note: In accordance with the NVIDIA documentation, utilization.gpu means “Percent of time over the past sample period during which a number of kernels was executing on the GPU. The sample period could also be between 1 second and 1/6 second depending on the product”.
Resource-based metrics are inherently retrospective as they only reflect usage after the resources have been consumed. They’re also capped at 100%, which presents an issue: when scaling based on these metrics, the utmost ratio for adjustment is usually the present utilization over the specified threshold (see scaling formula below). This ends in a conservative scale-up behavior that doesn’t necessarily match the actual demand of production traffic.
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
Request-based metrics
We turned to request-based metrics for more proactive signaling which might be also not capped at a 100%.
QPS is a widely known metric for its simplicity. Nonetheless, its application in generative AI, resembling with LLMs, continues to be an issue. QPS isn’t easy to configure and because of the variable cost per request, which is dependent upon the variety of tokens processed and generated, using QPS as a scaling metric can result in inaccuracies.
Concurrency, then again, has proven to be a great metric for reflecting the actual load on the system. It represents the variety of energetic requests either queued or being processed. This metric:
- Precisely reflects the load on the system. Little’s Law, which states that QPS multiplied by average latency equals concurrency, provides a sublime strategy to understand the connection between QPS and concurrency. In practice, the common latency per request is reasonably unknown in model serving. Nonetheless, by measuring concurrency, we don’t have to calculate average latency.
- Accurately calculate the specified replicas using the scaling formula. Allowing the deployment to directly scale to the perfect size without intermediate steps.
- Easy to configure based on batch size. For non-batchable models, it’s simply the variety of GPUs, since each can only handle one generation task at a time. For models that support batching, the batch size determines the concurrency level.
For concurrency to work, we want the support from the service framework to robotically instrument concurrency as a metric and serve it as a scaling signal for the deployment platform. We must also establish right scaling policies to assist against overzealous scale-up during a traffic spike or premature scale-down when traffic is sparse.
A one other necessary mechanism we integrated with concurrency is the request queue. It acts as a buffer and an orchestrator, ensuring that incoming requests are handled efficiently and without overloading any single server replica.
In a scenario with no request queue, all incoming requests are dispatched on to the server (6 requests within the image below). If multiple requests arrive concurrently, and there’s just one energetic server replica, it becomes a bottleneck. The server tries to process each request in a first-come-first-serve manner, often resulting in timeouts and a nasty client experience.
Conversely, with a request queue in place, the server consumes requests at an optimal rate, processing at a rate based on the concurrency defined for the service. When additional server replicas scale up, they too begin to drag from the queue. This mechanism prevents any single server from becoming overwhelmed and allows for a smoother, more manageable distribution of requests across the available infrastructure.
Our journey in exploring AI model scaling solutions has been an adventure, which has led us to ultimately create the scaling experience on BentoCloud — a platform that encapsulates all our learnings.
To avoid the impression of a promotion, we’ll illustrate our point with an image that’s value a thousand words. The monitoring dashboard below demonstrates the correlation between incoming requests and the scaling up of server instances.
Equally necessary to scaling up is the power to scale down. Because the requests waned to zero, the deployment reduced the variety of energetic instances accordingly. This ability ensures that no unnecessary costs are incurred for unused resources, aligning expenditure with actual usage.
We hope the takeaway is that scaling for model deployments must be considered a vital aspect of production applications. Unlike scaling CPU workloads, scaling model deployments on GPUs presents unique challenges, including cold start times, configuring scaling metrics, and orchestrating requests. When evaluating deployment platforms, their solutions to those challenges must be thoroughly assessed.