Using quantized models for memory-efficiency

Larger language models typically deliver superior performance but at the fee of reduced inference speed. For instance, Llama 2 70B significantly outperforms Llama 2 7B in downstream tasks, but its inference speed is roughly 10 times slower.

Many techniques and adjustments of decoding hyperparameters can speed up inference for very large LLMs. Speculative decoding, particularly, might be very effective in lots of use cases.

Speculative decoding uses a small LLM to generate the tokens that are then validated, or corrected if needed, by a significantly better and bigger LLM. If the small LLM is accurate enough, speculative decoding can dramatically speed up inference.

In this text, I first explain how speculative decoding works. Then, I show the way to run speculative decoding with different pairs of models involving Gemma, Mixtral-8x7B, Llama 2, and Pythia, all quantized. I benchmarked the inference throughput and memory consumption to focus on what configurations work one of the best.

Speculative decoding is presented by Google Research on this paper:

Fast Inference from Transformers via Speculative Decoding

It’s a quite simple and intuitive method. Nevertheless, as we are going to see intimately in the subsequent section, it’s also difficult to make it work.

Speculative decoding runs two models during inference: the major model we would like to make use of and a draft model. This draft model suggests the tokens during inference. Then, the major model checks the suggested tokens and corrects them if crucial. In the long run, the output of speculative decoding is similar because the one that may have generated the major model alone.

Here is an illustration of speculative decoding by Google Research:

This method can dramatically speed up inference if: