Optimizing machine learning models with dynamic shapes may be crucial for achieving higher performance and suppleness. Dynamic shapes discuss with the flexibility of a model to handle input data with various dimensions during runtime. Users utilize frameworks that support dynamic computation graphs, akin to TensorFlow’s eager execution or PyTorch. These frameworks allow constructing models that may adapt to variable input sizes during runtime.

There are various challenges in optimizing machine learning models with dynamic shapes, as many traditional optimizations rely on static shape evaluation. The missing information from dynamic dimensions can significantly affect the optimizations one can perform across operators and functions. Models with dynamic shapes have to handle various batch sizes. Optimizing for various batch sizes may be more difficult than optimizing for a set batch size, particularly in production settings.

Current machine learning (ML) compilers normally lower programs to hardware in a standard single-shot lowering flow, applying one optimization after the opposite, typically rewriting this system right into a lower-level representation. This approach often ends in losing shape and extra information between abstraction layers, making it harder to perform incremental optimizations across boundaries.

Researchers present Chill out. It’s a compiler abstraction for optimizing end-to-end dynamic machine learning workloads. It has first-class symbolic shape annotations to trace dynamic shape computations globally across this system. It also has a cross-level abstraction that encapsulates computational graphs, loop-level tensor programs, and library calls in a single representation to enable cross-level optimizations. It’s an end-to-end compilation framework to optimize dynamic shape models.

Researchers adopt a forward deduction method that deduces the annotation of an expression based on its input components. Forward deduction is straightforward and native, and one can obtain annotations for temporary variables during compiler passes. Moreover, when shapes can’t be inferred routinely, the forward deduction can use the outcomes of a user-inserted match forged to proceed inferring later annotations.

Researchers say all optimizations in Chill out are performed as composable dynamic shape–aware transformations. This incrementally optimizes or partially lowers portions of the computation using different approaches. It considers evaluation from other levels and incorporates further optimizations that assume dynamic shape relations.

Experimental results show that Chill out compiles and optimizes emerging LLMs onto diverse hardware backends, delivering competitive performance to heavily optimized platform-specific solutions. Moreover, Chill out supports LLMs on a broad set of devices and environments, including mobile phones, embedded devices, and web browsers through WebAssembly and WebGPU.

Try the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to affix our 32k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.

In the event you like our work, you’ll love our newsletter..

We’re also on Telegram and WhatsApp.