There are multiple ways to find out the associated fee of running a LLM (electricity use, compute cost, etc.), nevertheless, should you use a third-party LLM (a LLM-as-a-service) they typically charge you based on the tokens you utilize. Different vendors (OpenAI, Anthropic, Cohere, etc.) have alternative ways of counting the tokens, but for the sake of simplicity, we’ll consider the associated fee to be based on the variety of tokens processed by the LLM.
A very powerful a part of this framework is the concept that different models cost different amounts. The authors of the paper conveniently assembled the below table highlighting the difference in cost, and the difference between them is important. For instance, AI21’s output tokens cost an order of magnitude greater than GPT-4’s does on this table!
As an element of cost optimization we at all times have to determine a method to optimize the reply quality while minimizing the associated fee. Typically, higher cost models are sometimes higher performing models, able to provide higher quality answers than lower cost ones. The final relationship might be seen within the below graph, with Frugal GPT’s performance overlaid on top in red.
Using the vast cost difference between models, the researchers’ FrugalGPT system relies on a cascade of LLMs to provide the user a solution. Put simply, the user query begins with the most cost effective LLM, and if the reply is nice enough, then it’s returned. Nevertheless, if the reply shouldn’t be ok, then the query is passed along to the following most cost-effective LLM.
The researchers used the next logic: if a inexpensive model answers an issue incorrectly, then it is probably going that a costlier model will give the reply appropriately. Thus, to reduce costs the chain is ordered from least expensive to most costly, assuming that quality goes up as you get costlier.
This setup relies on reliably determining when a solution is nice enough and when it isn’t. To unravel for this, the authors created a DistilBERT model that may take the query and answer then assign a rating to the reply. Because the DistilBERT model is exponentially smaller than the opposite models within the sequence, the associated fee to run it is nearly negligible in comparison with the others.
One might naturally ask, if quality is most significant, why not only query the very best LLM and work on ways to cut back the associated fee of running the very best LLM?
When this paper got here out GPT-4 was the very best LLM they found, yet GPT-4 didn’t at all times give a greater answer than the FrugalGPT system! (Eagle-eyed readers will see this as a part of the associated fee vs performance graph from before) The authors speculate that just as probably the most capable person doesn’t at all times give the proper answer, probably the most complex model won’t either. Thus, by having the reply undergo a filtering process with DistilBERT, you’re removing any answers that aren’t as much as par and increasing the percentages of answer.
Consequently, this method not only reduces your costs but may also increase quality more so than simply using the very best LLM!
The outcomes of this paper are fascinating to think about. For me, it raises questions on how we will go even further with cost savings without having to take a position in further model optimization.
One such possibility is to cache all model answers in a vector database after which do a similarity search to find out if the reply within the cache works before starting the LLM cascade. This may significantly reduce costs by replacing a costly LLM operation with a relatively inexpensive query and similarity operation.
Moreover, it makes you wonder if outdated models can still be price cost-optimizing, as should you can reduce their cost per token, they’ll still create value on the LLM cascade. Similarly, the important thing query here is at what point do you get diminishing returns by adding recent LLMs onto the chain.