The best way to make your LLM do the fitting things, and do them right
Some time ago, I wrote the article Selecting the fitting language model in your NLP use case on Medium. It focussed on the nuts and bolts of LLMs — and while relatively popular, by now, I know it doesn’t actually say much about choosing LLMs. I wrote it at first of my LLM journey and one way or the other figured that the technical details about LLMs — their inner workings and training history — would speak for themselves, allowing AI product builders to confidently select LLMs for specific scenarios.
Since then, I even have integrated LLMs into multiple AI products. This allowed me to find how precisely the technical makeup of an LLM determines the ultimate experience of a product. It also strengthened the idea that product managers and designers must have a solid understanding of how an LLM works “under the hood.” LLM interfaces are different from traditional graphical interfaces. The latter provide users with a (hopefully clear) mental model by displaying the functionality of a product in a relatively implicit way. However, LLM interfaces use free text because the principal interaction format, offering way more flexibility. At the identical time, additionally they “hide” the capabilities and the restrictions of the underlying model, leaving it to the user to explore and discover them. Thus, an easy text field or chat window invites an infinite variety of intents and inputs and might display as many various outputs.
The responsibility for the success of those interactions just isn’t (only) on the engineering side — relatively, a giant a part of it needs to be assumed by whoever manages and designs the product. In this text, we are going to flesh out the connection between LLMs and user experience, working with two universal ingredients which you could use to enhance the experience of your product:
- Functionality, i.e., the tasks which might be performed by an LLM, comparable to conversation, query answering, and sentiment evaluation
- Quality with which an LLM performs the duty, including objective criteria comparable to correctness and coherence, but in addition subjective criteria comparable to an appropriate tone and elegance
(Note: These two ingredients are a part of any LLM application. Beyond these, most applications will even have a variety of more individual criteria to be fulfilled, comparable to latency, privacy, and safety, which is able to not be addressed here.)
Thus, in Peter Drucker’s words, it’s about “doing the fitting things” (functionality) and “doing them right” (quality). Now, as we all know, LLMs won’t ever be 100% right. As a builder, you may approximate the perfect experience from two directions:
- On the one hand, it’s essential strive for engineering excellence and make the fitting selections when choosing, fine-tuning, and evaluating your LLM.
- However, it’s essential work your users by nudging them towards intents covered by the LLM, managing their expectations, and having routines that fireside off when things go improper.
In this text, we are going to concentrate on the engineering part. The design of the perfect partnership with human users shall be covered in a future article. First, I’ll briefly introduce the steps within the engineering process — LLM selection, adaptation, and evaluation — which directly determine the ultimate experience. Then, we are going to take a look at the 2 ingredients — functionality and quality — and supply some guidelines to steer your work with LLMs to optimize the product’s performance along these dimensions.
A note on scope: In this text, we are going to consider the usage of stand-alone LLMs. Most of the principles and guidelines also apply to LLMs utilized in RAG (Retrieval-Augmented Generation) and agent systems. For a more detailed consideration of the user experience in these prolonged LLM scenarios, please consult with my book The Art of AI Product Development.
In the next, we are going to concentrate on the three steps of LLM selection, adaptation, and evaluation. Let’s consider each of those steps:
- LLM selection involves scoping your deployment options (particularly, open-source vs. industrial LLMs) and choosing an LLM whose training data and pre-training objective align along with your goal functionality. As well as, the more powerful the model you may select when it comes to parameter size and training data quantity, the higher the probabilities it should achieve a top quality.
- LLM adaptation via in-context learning or fine-tuning gives you the prospect to shut the gap between your users’ intents and the model’s original pre-training objective. Moreover, you may tune the model’s quality by incorporating the style and tone you desire to your model to assume into the fine-tuning data.
- LLM evaluation involves constantly evaluating the model across its lifecycle. As such, it just isn’t a final step at the tip of a process but a continuous activity that evolves and becomes more specific as you collect more insights and data on the model.
The next figure summarizes the method:
In real life, the three stages will overlap, and there will be back-and-forth between the stages. Basically, model selection is more the “one big decision.” In fact, you may shift from one model to a different further down the road and even should do that when latest, more suitable models appear in the marketplace. Nonetheless, these changes are expensive since they affect every part downstream. Past the invention phase, you is not going to intend to make them frequently. However, LLM adaptation and evaluation are highly iterative. They needs to be accompanied by continuous discovery activities where you learn more concerning the behavior of your model and your users. Finally, all three activities needs to be embedded right into a solid LLMOps pipeline, which is able to permit you to integrate latest insights and data with minimal engineering friction.
Now, let’s move to the second column of the chart, scoping the functionality of an LLM and learning how it may possibly be shaped throughout the three stages of this process.
You may be wondering why we talk concerning the “functionality” of LLMs. In any case, aren’t LLMs those versatile all-rounders that may magically perform any linguistic task we will consider? In actual fact, they’re, as famously described within the paper Language Models Are Few-Shot Learners. LLMs can learn latest capabilities from just a few examples. Sometimes, their capabilities will even “emerge” out of the blue during normal training and — hopefully — be discovered by likelihood. It is because the duty of language modeling is just as versatile because it is difficult — as a side effect, it equips an LLM with the flexibility to perform many other related tasks.
Still, the pre-training objective of LLMs is to generate the following word given the context of past words (OK, that’s a simplification — in auto-encoding, the LLM can work in each directions [3]). That is what a pre-trained LLM, motivated by an imaginary “reward,” will insist on doing once it’s prompted. Generally, there is sort of a spot between this objective and a user who involves your product to talk, get answers to questions, or translate a text from German to Italian. The landmark paper Climbing Towards NLU: On Meaning, Form, and Understanding within the Age of Data by Emily Bender and Alexander Koller even argues that language models are generally unable to get well communicative intents and thus are doomed to work with incomplete meaning representations.
Thus, it’s one thing to brag about amazing LLM capabilities in scientific research and exhibit them on highly controlled benchmarks and test scenarios. Rolling out an LLM to an anonymous crowd of users with different AI skills and intents—some harmful—is a unique form of game. This is particularly true when you understand that your product inherits not only the capabilities of the LLM but in addition its weaknesses and risks, and also you (not a third-party provider) hold the responsibility for its behavior.
In practice, now we have learned that it’s best to discover and isolate discrete islands of functionality when integrating LLMs right into a product. These functions can largely correspond to the several intents with which your users come to your product. For instance, it may very well be:
- Engaging in conversation
- Retrieving information
- Searching for recommendations for a selected situation
- Searching for inspiration
Oftentimes, these will be further decomposed into more granular, potentially even reusable, capabilities. “Engaging in conversation” may very well be decomposed into:
- Provide informative and relevant conversational turns
- Maintain a memory of past interactions (as a substitute of ranging from scratch at every turn)
- Display a consistent personality
Taking this more discrete approach to LLM capabilities provides you with the next benefits:
- ML engineers and data scientists can higher focus their engineering activities (Figure 2) on the goal functionalities.
- Communication about your product becomes on-point and specific, helping you manage user expectations and preserving trust, integrity, and credibility.
- Within the user interface, you should use a variety of design patterns, comparable to prompt templates and placeholders, to extend the probabilities that user intents are aligned with the model’s functionality.
Let’s summarize some practical guidelines to be certain that the LLM does the fitting thing in your product:
- During LLM selection, be certain you understand the essential pre-training objective of the model. There are three basic pre-training objectives (auto-encoding, autoregression, sequence-to-sequence), and every of them influences the behavior of the model.
- Many LLMs are also pre-trained with a complicated objective, comparable to conversation or executing explicit instructions (instruction fine-tuning). Choosing a model that’s already prepared in your task will grant you an efficient head start, reducing the quantity of downstream adaptation and fine-tuning it’s essential do to realize satisfactory quality.
- LLM adaptation via in-context learning or fine-tuning gives you the chance to shut the gap between the unique pre-training objective and the user intents you ought to serve.
- In the course of the initial discovery, you should use in-context learning to gather initial usage data and sharpen your understanding of relevant user intents and their distribution.
- In most scenarios, in-context learning (prompt tuning) just isn’t sustainable in the long run — it is just not efficient. Over time, you should use your latest data and learnings as a basis to fine-tune the weights of the model.
- During model evaluation, be certain to use task-specific metrics. For instance, Text2SQL LLMs (cf. this text) will be evaluated using metrics like execution accuracy and test-suite accuracy, while summarization will be evaluated using similarity-based metrics.
These are only short snapshots of the teachings we learned when integrating LLMs. My upcoming book The Art of AI Product Development comprises deep dives into each of the rules together with quite a few examples. For the technical details behind pre-training objectives and procedures, you may consult with this text.
Okay, so you might have gained an understanding of the intents with which your users come to your product and “motivated” your model to answer these intents. You may even have put out the LLM into the world within the hope that it should kick off the information flywheel. Now, if you ought to keep your good-willed users and acquire latest users, it’s essential quickly ramp up on our second ingredient, namely quality.
Within the context of LLMs, quality will be decomposed into an objective and a subjective component. The target component tells you when and why things go improper (i.e., the LLM makes explicit mistakes). The subjective component is more subtle and emotional, reflecting the alignment along with your specific user crowd.
Using language to speak comes naturally to humans. Language is ingrained in our minds from the start of our lives, and now we have a tough time imagining how much effort it takes to learn it from scratch. Even the challenges we experience when learning a foreign language can’t compare to the training of an LLM. The LLM starts from a blank slate, while our learning process builds on an incredibly wealthy basis of existing knowledge concerning the world and about how language works typically.
When working with an LLM, we should always continually remain aware of the numerous ways by which things can go improper:
- The LLM might make linguistic mistakes.
- The LLM might slack on coherence, logic, and consistency.
- The LLM may need insufficient world knowledge, resulting in improper statements and hallucinations.
These shortcomings can quickly turn into showstoppers in your product — output quality is a central determinant of the user experience of an LLM product. For instance, one in every of the most important determinants of the “public” success of ChatGPT was that it was indeed capable of generate correct, fluent, and comparatively coherent text across a big number of domains. Earlier generations of LLMs weren’t capable of achieve this objective quality. Most pre-trained LLMs which might be utilized in production today do have the aptitude to generate language. Nonetheless, their performance on criteria like coherence, consistency, and world knowledge will be very variable and inconsistent. To attain the experience you’re aiming for, it is vital to have these requirements clearly prioritized and choose and adapt LLMs accordingly.
Venturing into the more nuanced subjective domain, you ought to understand and monitor how users feel around your product. Do they feel good and trustful and get right into a state of flow once they use it? Or do they go away with feelings of frustration, inefficiency, and misalignment? Numerous this hinges on individual nuances of culture, values, and elegance. When you are constructing a copilot for junior developers, you hardly want it to talk the language of senior executives and vice versa.
For the sake of example, imagine you’re a product marketer. You’ve spent quite a lot of your time with a fellow engineer to iterate on an LLM that helps you with content generation. Sooner or later, you end up chatting with the UX designer in your team and bragging about your latest AI assistant. Your colleague doesn’t get the necessity for therefore much effort. He’s frequently using ChatGPT to help with the creation and evaluation of UX surveys and may be very satisfied with the outcomes. You counter — ChatGPT’s outputs are too generic and monotonous in your storytelling and writing tasks. In actual fact, you might have been using it at first and got quite embarrassed because, sooner or later, your readersstarted to acknowledge the characteristic ChatGPT flavor. That was a slippery episode in your profession, after which you made the decision you needed something more sophisticated.
There isn’t a right or improper on this discussion. ChatGPT is sweet for straightforward factual tasks where style doesn’t matter that much. Against this, you as a marketer need an assistant that may assist in crafting high-quality, persuasive communications that talk the language of your customers and reflect the unique DNA of your organization.
These subjective nuances can ultimately define the difference between an LLM that’s useless because its outputs have to be rewritten anyway and one which is “adequate” so users start using it and feed it with suitable fine-tuning data. The holy grail of LLM mastery is personalization — i.e., using efficient fine-tuning or prompt tuning to adapt the LLM to the person preferences of any user who has spent a certain quantity of time with the model. When you are only starting out in your LLM journey, these details may appear far off — but in the long run, they will make it easier to reach a level where your LLM delights users by responding in the precise manner and elegance that’s desired, spurring user satisfaction and large-scale adoption and leaving your competition behind.
Listed here are our suggestions for managing the standard of your LLM:
- Be alert to different sorts of feedback. The search for quality is continuous and iterative — you begin with a couple of data points and a really rough understanding of what quality means in your product. Over time, you flesh out increasingly details and learn which levers you may pull to enhance your LLM.
- During model selection, you continue to have quite a lot of discovery to do — start with “eyeballing” and testing different LLMs with various inputs (ideally by multiple team members).
- Your engineers will even be evaluating academic benchmarks and evaluation results which might be published along with the model. Nonetheless, be mindful that these are only rough indicators of how the model will perform in your specific product.
- Initially, perfectionism isn’t the reply. Your model needs to be just adequate to draw users who will start supplying it with relevant data for fine-tuning and evaluation.
- Bring your team and users together for qualitative discussions of LLM outputs. As they use language to guage and debate what is correct and what’s improper, you may step by step uncover their objective and emotional expectations.
- Make sure that to have a solid LLMOps pipeline in place so you may integrate latest data easily, reducing engineering friction.
- Don’t stop — at later stages, you may shift your focus toward nuances and personalization, which will even make it easier to sharpen your competitive differentiation.
Pre-trained LLMs are highly convenient — they make AI accessible to everyone, offloading the large engineering, computation, and infrastructure spending needed to coach an enormous initial model. Once published, they’re able to use, and we will plug their amazing capabilities into our product. Nonetheless, when using a third-party model in your product, you inherit not only its power but in addition the numerous ways by which it may possibly and can fail. When things go improper, the last item you ought to do to take care of integrity is responsible an external model provider, your engineers, or — worse — your users.
Thus, when constructing with LLMs, it is best to not only search for transparency into the model’s origins (training data and process) but in addition construct a causal understanding of how its technical makeup shapes the experience offered by your product. It will permit you to find the sensitive balance between kicking off a strong data flywheel at first of your journey and constantly optimizing and differentiating the LLM as your product matures toward excellence.
[1] Janna Lipenkova (2022). Selecting the fitting language model in your NLP use case, Medium.
[2] Tom B. Brown et al. (2020). Language Models are Few-Shot Learners.
[3] Jacob Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
[4] Emily M. Bender and Alexander Koller (2020). Climbing towards NLU: On Meaning, Form, and Understanding within the Age of Data.
[5] Janna Lipenkova (upcoming). The Art of AI Product Development, Manning Publications.
Note: All images are by the creator, except when noted otherwise.