Start with multimodal conversational models using the open-source LLaVA model.
Large Language Models have proved themselves to be a revolutionary technology. Quite a few applications exploiting their capabilities have been already developed and plenty of more are expected to return soon. Probably the most interesting applications of Large Language Models is their deployment as intelligent assistants in a position to help human users in a wide range of tasks. Chat models trained with instruction tuning and Reinforcement Learning from Human Feedback (RLHF) have shown very promising capabilities of following human instructions and carrying out the assigned tasks. Nonetheless, they’re limited of their applicability to language-only tasks.
Multimodal conversational models aim to unleash the facility of Large Language Models to tackle problems that require combining natural language with other modalities to be solved. Specifically, vision-language models have received increasing attention for the reason that introduction of vision capabilities to GPT-4V. Empowering the natural language capabilities of GPT-4 with image understanding has led to a robust chat assistant that may also help users with tasks requiring each vision and language understanding. While the vision capabilities of GPT-4V are impressive, closed-source models limit the potential for research and experimentation with this amazing technology. Fortunately, some open-source models appeared bringing the facility of vision language models to the community in an easily accessible and transparent way. These models also proceed the trend of increased concentrate on computing and memory efficiency, a trend already seen for open-source Large Language Models. That is a crucial feature since it facilitates the widespread adoption of those models.
On this tutorial, I’ll walk through the means of making a vision chat assistant using the LLaVA (Large Language and Vision Assistant) model introduced within the Visual Instruction Tuning paper. I’ll first give a transient introduction to the LLaVA model and its improvements before discussing a straightforward code implementation of a vision chat assistant using the code provided within the official repository. I’ll then present some examples I crafted to showcase the capabilities and limitations of the model.
LLaVA
The LLaVA model was introduced within the paper Visual Instruction Tuning, after which further improved in Improved Baselines with Visual Instruction Tuning (also known as LLaVA-1.5). The thought behind it’s to extract visual embeddings from a picture and treat them in the identical way as embeddings coming from language tokens by feeding them to a Large Language Model. Intuitively, we will think that the image shall be described with “words” that the language model will use to generate its answer. To decide on the correct “words” the model uses a pre-trained CLIP visual encoder to extract the visual embeddings after which projects them into the word embedding space of the language model. The latter operation is completed with a vision-language connector, which was originally chosen to be a straightforward linear layer in the primary paper Visual Instruction Tuning, and later replaced with a more expressive Multilayer Perceptron (MLP) in Improved Baselines with Visual Instruction. The architecture of the model is depicted below.
One in all the benefits of the tactic is that through the use of a pre-trained vision encoder and a pre-trained language model, only the vision-language connector (which is a light-weight module) should be learned from scratch. Specifically, the training of LLava consists of two stages:
- Pre-training for feature alignment: each the pre-trained vision encoder and language model are frozen, and only the weights of the vision-language connector are updated. All training samples consist of text-image pairs packed right into a single-turn conversation. This stage goals to coach the vision-language connector to align the embeddings of the vision encoder with the text embeddings of the language model.
- Nice-tuning with visual instructions: on this stage, only the weights of the vision encoder are frozen while the vision-language connector and the language model are fine-tuned together. The model is fine-tuned on image-based instruction-following tasks. It’s interesting to note that a few of this data has been created through the use of language-only GPT4 to create instruction-following samples from the caption of the photographs and the coordinates of the bounding boxes of the entities depicted.
Making a vision chatbot using the code provided within the official repository is fairly easy. The repository also provides standardized chat templates that will be used to parse the inputs in the correct format. Following the correct format utilized in training is crucial for the standard of the reply generated by the model. The precise template is dependent upon the language model used. The template for LLaVA-1.5 with a pre-trained Vicuna language model will seem like this:
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER: User's prompt
ASSISTANT: Assistant answer
USER: One other prompt
The primary few lines are the overall system prompt utilized by the model. The special tokens
The chatbot will be defined in only one easy Python class.
class LLaVAChatBot:
def __init__(self,
model_path: str = 'liuhaotian/llava-v1.5-7b',
device_map: str = 'auto',
load_in_8_bit: bool = True,
**quant_kwargs) -> None:
self.model = None
self.tokenizer = None
self.image_processor = None
self.conv = None
self.conv_img = None
self.img_tensor = None
self.roles = None
self.stop_key = None
self.load_models(model_path,
device_map=device_map,
load_in_8_bit=load_in_8_bit,
**quant_kwargs)def load_models(self, model_path: str,
device_map: str,
load_in_8_bit: bool,
**quant_kwargs) -> None:
"""Load the model, processor and tokenizer."""
quant_cfg = BitsAndBytesConfig(**quant_kwargs)
self.model = LlavaLlamaForCausalLM.from_pretrained(model_path,
low_cpu_mem_usage=True,
device_map=device_map,
load_in_8bit=load_in_8_bit,
quantization_config=quant_cfg)
self.tokenizer = AutoTokenizer.from_pretrained(model_path,
use_fast=False)
vision_tower = self.model.get_vision_tower()
vision_tower.load_model()
vision_tower.to(device='cuda')
self.image_processor = vision_tower.image_processor
disable_torch_init()
def setup_image(self, img_path: str) -> None:
"""Load and process the image."""
if img_path.startswith('http') or img_path.startswith('https'):
response = requests.get(img_path)
self.conv_img = Image.open(BytesIO(response.content)).convert('RGB')
else:
self.conv_img = Image.open(img_path).convert('RGB')
self.img_tensor = self.image_processor.preprocess(self.conv_img,
return_tensors='pt'
)['pixel_values'].half().cuda()
def generate_answer(self, **kwargs) -> str:
"""Generate a solution from the present conversation."""
raw_prompt = self.conv.get_prompt()
input_ids = tokenizer_image_token(raw_prompt,
self.tokenizer,
IMAGE_TOKEN_INDEX,
return_tensors='pt').unsqueeze(0).cuda()
stopping = KeywordsStoppingCriteria([self.stop_key],
self.tokenizer,
input_ids)
with torch.inference_mode():
output_ids = self.model.generate(input_ids,
images=self.img_tensor,
stopping_criteria=[stopping],
**kwargs)
outputs = self.tokenizer.decode(
output_ids[0, input_ids.shape[1]:]
).strip()
self.conv.messages[-1][-1] = outputs
return outputs.rsplit('', 1)[0]
def get_conv_text(self) -> str:
"""Return full conversation text."""
return self.conv.get_prompt()
def start_new_chat(self,
img_path: str,
prompt: str,
do_sample=True,
temperature=0.2,
max_new_tokens=1024,
use_cache=True,
**kwargs) -> str:
"""Start a brand new chat with a brand new image."""
conv_mode = "v1"
self.setup_image(img_path)
self.conv = conv_templates[conv_mode].copy()
self.roles = self.conv.roles
first_input = (DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN +
DEFAULT_IM_END_TOKEN + 'n' + prompt) # f"{self.roles[0]}: {prompt}")
self.conv.append_message(self.roles[0], first_input)
self.conv.append_message(self.roles[1], None)
if self.conv.sep_style == SeparatorStyle.TWO:
self.stop_key = self.conv.sep2
else:
self.stop_key = self.conv.sep
answer = self.generate_answer(do_sample=do_sample,
temperature=temperature,
max_new_tokens=max_new_tokens,
use_cache=use_cache,
**kwargs)
return answer
def continue_chat(self,
prompt: str,
do_sample=True,
temperature=0.2,
max_new_tokens=1024,
use_cache=True,
**kwargs) -> str:
"""Proceed the present chat."""
if self.conv is None:
raise RuntimeError("No existing conversation found. Start a brand new"
"conversation using the `start_new_chat` method.")
self.conv.append_message(self.roles[0], prompt)
self.conv.append_message(self.roles[1], None)
answer = self.generate_answer(do_sample=do_sample,
temperature=temperature,
max_new_tokens=max_new_tokens,
use_cache=use_cache,
**kwargs)
return answer
In the event you are acquainted with the transformers library, you’ll recognize most of the usual features, and the operations performed must be straightforward to grasp. Let’s go quickly over the methods of the LLaVAChatBot class defined above.
- load_models: this method loads the language models, the tokenizer, and the image processor with the required parameters for quantization using the BitsAndBytes library. The code shadows the from_pretrained method utilized by Hugging Face transformers models. BitsAndBytes allows quantizing to model to 8bit or 4bit for reduced GPU memory requirements.
- setup_image: it loads the image from a neighborhood path or a URL and converts it to a tensor using the image processor.
- generate_answer: this method returns the model’s answer continuing the present conversation concerning the provided image. Again the generate approach to the LLaVa model is analogous to the generate approach to Hugging Face transformers models.
- get_conv_text: this method returns the raw text of the conversation to this point.
- start_new_chat: that is one in all the 2 most important methods of the chatbot, it’s used to begin a brand new chat with the model. It creates a brand new conversation given the image and the initial prompt to pass to the model. It takes care of establishing the conversation using the templates defined within the repository following the format discussed within the previous section.
- continue_chat: the opposite most important method, it continues an existing conversation about a picture.
I actually have provided the complete code on this Colab notebook so that you would be able to try it yourself.
On this section, I collect some interesting examples obtained while testing the capabilities of the model. All of the examples on this post were created using the model llava-v1.5–7b with 8-bit quantization.
For a start, let’s take a look at the model’s capabilities to explain and understand images. Below I asked the model to explain a picture picturing a white tiger.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER:
Describe the image in details.
ASSISTANT: The image incorporates a white tiger laying down on a rocky surface,
possibly a stone wall or a rock. The tiger appears to be relaxed and
comfortable in its environment. The tiger's position is such that it covers a
significant slice of the image, from the left side to almost the whole right
side. The rocky surface provides a natural and rugged backdrop for the scene.
USER: What is exclusive about this tiger?
ASSISTANT: This tiger is exclusive since it is a white tiger, which is an
extremely rare and visually striking variation of the species. White tigers are
born with a genetic mutation that causes their fur to be predominantly white,
with the remainder of their body being a mixture of orange and black stripes. This
distinctive appearance sets them other than the more common orange and black
striped tigers.
The model seems to have the option to explain the image intimately and likewise to reason concerning the subject of the image, accurately assessing that a white tiger is a rare occurrence.
Next, I tested if the model is in a position to provide the user with actionable instructions to perform various tasks. That is what distinguishes a “captioning” model from a real vision-chat assistant. For instance, can the model provide recipes from the ingredients present in a picture? The result’s reported below.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions.
USER:
What can I cook with these ingredients? ASSISTANT: With the ingredients displayed on the table, you may prepare a
delicious roast dinner. The meat, which is a big piece of beef, will be
seasoned and cooked within the oven, while the vegetables, including carrots,
potatoes, and onions, will be roasted alongside the meat. You can even prepare a
side dish with the potatoes, carrots, and onions, which will be served alongside
the roast dinner. The presence of a bowl of sauce suggests that you would be able to use it
as a dipping sauce or to boost the flavors of the roast and vegetables.
Overall, this mixture of ingredients offers a hearty and flavorful meal
that will be enjoyed with family and friends.
USER: Could the recipe you suggested be included in a vegetarian weight-reduction plan?
ASSISTANT: The recipe I suggested, which involves roasting a big piece of
beef and vegetables, is just not suitable for a vegetarian weight-reduction plan. The meat is the
most important protein source within the dish, and it is just not suitable for vegetarians who
avoid consuming meat. Nonetheless, you may still create a vegetarian-friendly meal
using the vegetables and sauce availableon the table. For instance, you may
prepare a vegetable roast with the carrots, potatoes, and onions, and serve it
alongside a side of mashed potatoes or a vegetable-based sauce. This manner, you
can still benefit from the flavors and textures of the dish without including meat.
USER: Are you able to suggest a vegetarian recipe with minimal modifications of the
depicted ingredients?
ASSISTANT: Definitely! Here's a vegetarian recipe that will be made using the
ingredients available on the table:
Roasted Vegetable Medley with Sauce
Ingredients:
* Carrots
* Potatoes
* Onions
* Olive oil
* Salt
* Pepper
* Herbs (akin to thyme, rosemary, or parsley)
* Sauce (optional)
Instructions:
1. Preheat your oven to 400°F (200°C).
2. Wash and peel the carrots, potatoes, and onions. Cut them into bite-sized
pieces.
3. In a big bowl, toss the vegetables with olive oil, salt, and pepper.
4. Spread the vegetables evenly on a baking sheet lined with parchment paper.
5. Roast the vegetables within the preheated oven for 20-25 minutes, or until they
are tender and barely browned.
6. If desired, prepare a sauce by combining ingredients akin to tomatoes,
garlic, and herbs. You should utilize this sauce as a dipping sauce or to boost the
flavors of the roasted vegetables.
7. Serve the roasted vegetable medley alongside a side of mashed potatoes or a
vegetable-based sauce.
This vegetarian recipe lets you benefit from the flavors and textures of the
roasted vegetables without including meat. The sauce will be easily customized
to fit your taste preferences, and the dish will be served alongside a spread
of side dishes, akin to mashed potatoes or a vegetable-based sauce.
The model seems to have a great understanding of the ingredients present within the image and additionally it is able to connecting the person words to more abstract concepts (i.e. a meal with meat is just not vegetarian).
Let’s now see how the model will be used to get pieces of recommendation. This can be a useful feature to get recent ideas when brainstorming and might provide real value to the user. Within the conversation below, I asked the model for suggestions to vary the look of a café.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER:
How can I renovate this café to make it look more vintage?
ASSISTANT: To make the café look more vintage, you may consider the next
changes:
1. Update the lighting: Replace the present lighting with vintage-style lamps
or lanterns, which is able to create a warm and comfortable atmosphere.
2. Add vintage decorations: Incorporate antique items, akin to vintage clocks,
artwork, or decorative plates, to create a nostalgic ambiance.
3. Use vintage furniture: Replace the present stools with vintage-style chairs
or barstools, which might add character and charm to the space.
4. Incorporate greenery: Add more greenery, akin to potted plants or vines, to
create a lush and alluring atmosphere.
5. Update the bar: Replace the present bar with a vintage-style bar, complete
with a vintage-inspired backsplash and countertop.
6. Use vintage-style glassware: Replace the present glassware with vintage-style
glasses, akin to wine glasses, beer mugs, or cocktail shakers.
7. Add vintage-style signage: Replace the present signage with vintage-style
menu boards or chalkboards, which might add a touch of nostalgia to the space.
8. Use vintage-style wallpaper or paint: Apply a vintage-style wallpaper or
paint to the partitions, which might create a singular and charming atmosphere.
By implementing these changes, the café will appear more vintage and alluring,
making a memorable experience for its patrons.
The recommendation looks on point, although quite generic, identifying the present elements that could possibly be modified to provide a more vintage style.
Let’s proceed with something practical that may aid the users of their day by day activities. Can the model help with writing advertisements or listings?
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER:
Write an advertisment for this chair
ASSISTANT: Introducing the trendy and trendy black and wood chair, perfect for
any contemporary setting. This chair incorporates a sleek black seat and a wood
base, making a striking contrast against any background. Its minimalist
design makes it a flexible addition to any room, whether it's used as a dining
chair, a side table, or an announcement piece in a living space. The chair's sturdy
construction ensures it may well withstand day by day use, while its clean lines and
sleek appearance add a touch of sophistication to any room. Upgrade your living
space with this stylish and functional chair today.
USER: Write now an commercial for a similar chair but targeting cafes
ASSISTANT: Attention café owners and design enthusiasts! Introducing the black
and wood chair, the right addition to your café's seating area. This stylish
and functional chair offers a contemporary touch to your establishment, with its
sleek black seat and wood base making a striking contrast against any
background. Its minimalist design makes it a flexible selection for any café
setting, whether it's used as a dining chair, a side table, or an announcement
piece in a lounge area. The chair's sturdy construction ensures it may well
withstand day by day use, while its clean lines and sleek appearance add a touch of
sophistication to any room. Upgrade your café's ambiance with this stylish and
functional chair today.
The model is in a position to pinpoint the features of the chair accurately and creates an appealing commercial, accurately customizing the reply to different targets when asked to.
As a final test, allow us to see if prompting techniques which have been demonstrated to be helpful for Large Language Models will be used with LLaVA as well. Below is an example of attempting zero-shot Chain of Thought prompting.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER:
Would the fish depicted within the image float if thrown in water? Think step by
step.
ASSISTANT: The fish depicted within the image is a big white fish sculpture or
statue, which is lying on the bottom on a brick walkway near a body of water.
If thrown into the water, the fish wouldn't float since it is a solid,
inanimate object. It will sink to the underside of the water, because it is just not
buoyant resulting from its composition and size. The fish's position on the bottom and
its lack of buoyancy make it unimaginable for it to drift in water.
USER: What material could possibly be used to make a fish sculpture that floats on
water?
ASSISTANT: To make a fish sculpture that floats on water, one could use a
lightweight, buoyant material akin to expanded polystyrene foam or polyethylene
foam. These materials are commonly used for making buoyant objects like life
jackets, flotation devices, and fishing floats. By shaping the froth into the
desired fish form, it could have the option to drift on the water's surface,
simulating the behavior of an actual fish.
The model is in a position to explain why the fish wouldn’t float, accurately identifying that it is just not an actual fish. It also follows the instruction to think step-by-step before giving the ultimate answer. For comparison, here is an example of the conversation without the “Think step-by-step” prompt.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions.USER:
Would the fish depicted within the image float if thrown in water?
ASSISTANT: No, the fish depicted within the image wouldn't float if thrown in
water. It's a large, white, stone fish sculpture, which is just not an actual fish
and doesn't have the buoyancy properties of a natural fish.
While the ultimate answer remains to be correct, the reason given on this case is far less detailed and the reply is given at the beginning of the sentence. For the reason that generative language model utilized by LLaVA is causal, which means the provided answer doesn’t depend on the following explanation. It will be interesting to perform more extensive experiments to check if Chain of Thought (CoT) can, typically, improve vision-language models’ performance in complex reasoning tasks much like what has been observed for Large Language Models in Chain-of-Thought Prompting Elicits Reasoning in Large Language Models and Large Language Models are Zero-Shot Reasoners.
While very powerful, LLaVA comes with its limitations. For instance, the model has been trained to make use of just one image per chat so it isn’t in a position to handle more complex conversations that need interactions with multiple images. It’s value noticing, nevertheless, that this limitation is just not intrinsic to the architecture of the model and will be amended by including multi-image chat data during training.
Throughout the tests, I noticed that the model (not less than the 8bit quantized version of the 1.5 7b model) has trouble with counting elements in a picture and typically with handling numerical tasks. Below is an example.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER:
What number of automotive are contained in the parking zone?
ASSISTANT: There are several cars parked contained in the parking zone.
USER: List the numbers of the parking spots occupied by cars
ASSISTANT: The parking spots occupied by cars are 66, 67, 68, 69, 70, and 71.
The model, like all Large Language Models, can be susceptible to hallucinations and will be easily tricked into making mistakes using appropriate prompts. For instance, if supplied with an image of the desert and asked what species is the fish within the image, it hallucinates a desert fish.
A chat between a curious user and a man-made intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user's questions. USER:
Tell me the species of the fish on this image
ASSISTANT: The fish on this image is a desert fish, which is a species adapted
to living in desert environments.
LLaVA shows impressive capabilities in vision-language understanding. It marks a transparent step forward for multimodal open-source vision-language models. One in all the most important benefits of LLaVA is that it’s lightweight to coach and fine-tune. For example, the complete training of LLaVA 1.5 13b took just one.2M data and roughly 1 day on a single 8-A100 node. This makes it suitable for fine-tuning on specific domains to get an authority assistant, as was done, for instance, in LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day.
Adding vision capabilities to talk assistants expands the realm of applications of such models, bringing their revolutionizing potential to more complex and nuanced tasks. Treating image features as language tokens also brings up the opportunity of using all of the advanced prompting techniques used with text-only language models and further expands them. For instance, one could expand the facility of Retrieval Augmented Generation by retrieving each texts and pictures which are relevant to the conversation. In actual fact, using the shared image-text embedding space of CLIP it is feasible to retrieve each external documents and external images starting with either an input text or picture!
One other interesting direction to expand the capabilities of the model is presented in LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing. The most important idea is to mix the varied capabilities of vision-language chat models, text-to-image generative models, and other vision models (akin to image segmentation models) to get an assistant able to handling multimodal inputs and generating multimodal outputs.
In conclusion, LLaVA marked a crucial step for open-source multimodal generative models, which have shown impressive capabilities and are attracting loads of interest. With the more widespread adoption of open-source models, I consider we’ll soon witness a rapid increase in recent applications of those powerful models.
Thanks for reading! If you need to check out the code yourself you may take a look at this Colab notebook.