Utilize SageMaker Pipelines, JumpStart, and Make clear to Effective-Tune and Evaluate a Llama 7B Model

2023 was the 12 months that witnessed the rise of assorted Large Language Models (LLMs) within the Generative AI space. LLMs have incredible power and potential, but productionizing them has been a consistent challenge for users. An especially prevalent problem is what LLM should one use? Much more specifically, how can one evaluate an LLM for accuracy? This is particularly difficult when there’s a lot of models to pick from, different datasets for fine-tuning/RAG, and quite a lot of prompt engineering/tuning techniques to think about.

To resolve this problem we want to determine DevOps best practices for LLMs. Having a workflow or pipeline that might help one evaluate different models, datasets, and prompts. This field is beginning to get often called LLMOPs/FMOPs. Among the parameters that will be considered in LLMOPs are shown below, in a (extremely) simplified flow:

In this text, we’ll attempt to tackle this problem by constructing a pipeline that fine-tunes, deploys, and evaluates a Llama 7B model. You can even scale this instance, through the use of it as a template to match multiple LLMs, datasets, and prompts. For this instance, we’ll be utilizing the next tools to construct this pipeline:

• SageMaker JumpStart: SageMaker JumpStart provides various FM/LLMs out of the box for each fine-tuning and deployment. Each these processes will be quite complicated, so JumpStart abstracts out the specifics and allows you to specify your dataset and model metadata to conduct fine-tuning and deployment. On this case we select Llama 7B and conduct Instruction fine-tuning which is supported out of the box. For a deeper introduction into JumpStart fine-tuning please confer with this blog and this Llama code sample, which we’ll use as a reference.
• SageMaker Make clear/FMEval: SageMaker Make clear provides a Foundation Model Evaluation tool via the SageMaker Studio UI and the open-source Python FMEVal library. The feature comes built-in with quite a lot of different algorithms spanning different NLP…