Part 1 — Customized Prompts via Language Models (CuPL)
Unimodal models are designed to work with data from a single mode, which might be either text or images. These models focus on understanding and generating content specific to their chosen mode. For instance, GPT are excellent at generating human-like text. They’ve been used for tasks like language translation, text generation, and answering questions. Convolutional Neural Networks (CNNs) are examples of image models that excel at tasks like image classification, object detection, and image generation. Currently, many interesting tasks comparable to Visual Query Answering (VQA) and Image-Text retrieval etc. require multimodal capabilities. Is it possible to mix each text and image processing? We will! CLIP stands out as one among the initial highly successful image-text models, demonstrating proficiency in each image recognition and text comprehension.
We’ll divide this text into the next sections:
- Introduction
- Architecture
- Training process and Contrastive loss
- Zero-shot capability
- CuPL
- Conclusions
The CLIP model is a formidable zero-shot predictor, enabling predictions on tasks it hasn’t explicitly been trained for. As we are going to see more intimately in the subsequent sections, by utilizing natural language prompts to question images, CLIP can perform image classification without requiring task-specific training data. Nevertheless, its performance might be significantly enhanced with just a few tricks. On this series of articles, we are going to explore methods that leverage additional prompts generated by Large Language Models (LLM) or a few-shot training examples without involving any parameter training. These approaches offer a definite advantage as they’re computationally less demanding and don’t necessitate fine-tuning additional parameters.
CLIP is a dual encoder model with two separate encoders for visual and textual modalities that encode images and texts independently. Such architecture is different from the fusion encoder that allows the interaction between visual and textual modalities through cross-attention which involves learning attention weights that help the model concentrate on specific regions of…