Yow will discover the code on this GitHub repo:
https://github.com/amirarsalan90/personal_llm_assistant

The essential components of the app include:

Llama-cpp-python is a python binding for the nice llama.cpp , which implements many Large Language Models in C/C++ . Due to its wide adoption by open-source community, I made a decision to make use of it on this tutorial.

Note: I even have tested this app on a system with Nvidia RTX4090 gpu.

Very first thing first, lets create a brand new conda environment:

conda create --name assistant python=3.10
conda activate assistant

Next we want to put in llama-cpp-python. As mentioned in llama-cpp-python descriptions, llama.cpp supports quite a few hardware acceleration backends to hurry up inference. As a way to leverage the GPU and run the LLM on GPU, we’ll construct this system with CUBLAS. I had some issues with attending to offload the model on GPU and I finally found this post on properly install:

export CMAKE_ARGS="-DLLAMA_CUBLAS=on"
export FORCE_CMAKE=1
pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir
pip install llama-cpp-python[server]