With the recent enhancement of visual instruction tuning methods, Multimodal Large Language Models (MLLMs) have demonstrated remarkable general-purpose vision-language capabilities. These capabilities make them key constructing blocks for contemporary general-purpose visual assistants. Recent models, including MiniGPT-4, LLaVA, InstructBLIP, and others, exhibit impressive visual reasoning and instruction-following abilities. Although a majority of them depend on image-text pairs for image-level vision-language alignment, they perform well on this domain. Nonetheless, their reliance on box-level and image-level understanding is the first reason MLLMs fall short in replicating their performance on fine-grained vision-language alignment tasks on the pixel level. Moreover, the limited availability of mask-based instruction data for training poses challenges in further enhancing MLLMs.

Osprey is a mask-text instruction training method with the first aim of extending MLLMs. It incorporates fine-grained masked regions in language instruction to realize pixel-level visual-language understanding. To perform this, the Osprey framework curates a mask-based region-text dataset with over 700 thousand samples. It injects pixel-level representation into Large Language Models (LLMs) to design a vision-language model. Notably, the Osprey framework adopts a convolutional CLIP model as its vision encoder and integrates a mask-aware visual extractor into its architecture. This permits for precise extraction of visual mask features from high-resolution input.

In this text, we’ll discuss the Osprey framework and delve deeper into its architecture. We can even explore the curated region-text dataset with over 700 thousand samples and compare its performance in various region understanding tasks. So, let’s start.

Multimodal Large Language Models like MiniGPT-4, Otter, Qwen-LV, InstructBLIP and others are the frontrunners for developing general-purpose visual assistants, they usually are renowned for his or her exceptional multimodal and vision generative capabilities. Nonetheless, Multimodal Large Language Models suffer from a significant challenge as they deliver unsatisfactory results on fine-grained image understanding tasks like captioning, region classification, and reasoning. A serious reason for the sub-par performance on fine-grained image understanding tasks is the dearth of alignment at region-level. Recent MLLMs like GPT4RoI, Shikra and others aim to enable region-level understanding in vision-language models by processing bounding-box specified regions, and leveraging visual instruction tuning with spatial features at object-level. 

Although the approach to enable region-level understanding might improve the performance, employing sparse bounding boxes because the referring input region directly might introduce irrelevant background features resulting in inaccurate region-text pair alignment for visual instruction tuning on large language models. In the course of the inference process, the box-level referring input won’t find a way to detect & represent the thing precisely; which may end in semantic deviation as demonstrated in the next image. 

As compared, using fine-grained masks as an alternative of coarse bounding boxes because the referring input might find a way to represent objects with more precision. Recently developed SAM or Segment Anything Model trains on billions of high-quality masks, demonstrates remarkable segmentation quality on zero-shot objects and supports the usage of points or easy bounding boxes as prompts. Nonetheless, the SAM framework cannot generate primary semantic labels, nor can they supply detailed semantic captions and attributes. In consequence, existing models lack inherent multimodal fine-grained information, and have a limited-understanding of scenes within the real-world. 

To tackle the challenges faced by the prevailing MLLMs, Osprey, a novel mask-text instruction training method goals to increase the capabilities of multimodal large language models for fine-grained understanding on pixel-level. The Osprey framework introduces a mask-aware visual extractor that captures visual mask features with various granularity precisely. The framework then interleaves the visual features with language instructions to generate the input sequence for the big language model, and leverages convolutional CLIP architecture to facilitate the usage of high resolution input. Owing to its design and architecture, the Osprey framework is in a position to achieve fine-grained semantic understanding for object-level and part-level regions, and provides detailed object attributes together with primary object category and enhanced descriptions of complex scenes. 

By leveraging the capabilities of visual instruction tuning, the Osprey framework enables latest capabilities beyond image-level and box-level understanding of the scenes because the Osprey framework can generate fine-grained semantics using class-agnostic masks from off the shelf SAMs. Moreover, Osprey also shows remarkable capabilities across referring object classification, open-vocabulary recognition, regional-level captioning, and detailed region description tasks. 

Osprey : Methodology and Architecture

The next figure demonstrates the architecture overview of the Osprey framework consisting of a big language model, pixel-level mask aware visual extractor, and an image-level vision encoder. 

For a given image, the input language, and the referring mask regions, the framework performs conversion and tokenization to generate embeddings before sending the language embedding sequences and interleaved mask features to the big language model to acquire fine-grained semantic understandings.

Convolutional CLIP Vision Encoder

The vision encoder deployed in a majority of multimodal large language models is exemplified using a ViT-based CLIP model. In consequence, the framework adopts a picture resolution of either 224×224 pixels or 336 x 336 pixels. Nonetheless, the usage of the ViT-based CLIP model makes it difficult for the model to realize fine-grain image understanding of pixel-level representations, an issue amplified further in small regions. Moreover, the computational overload related to the ViT architecture hinders the opportunity of increasing the input image resolution. 

To tackle the challenge, the Osprey framework implements a convolutional CLIP model because the vision encoder in its architecture. Traditionally, Convolutional Neural Networks based CLIP models have demonstrated remarkable generalization capabilities across different input resolutions when put against vision transformer based CLIP models. Implementing a CNN-based CLIP model makes room for fast inference and efficient training without compromising on the model’s performance. Moreover, a  CNN-based CLIP model is able to generating multi-scale feature maps that the framework then directly uses for feature extraction in each subsequent object region. 

Mask Aware Visual Extractor

In contrast to existing region-based models that use sparse bounding boxes because the referring input, the Osprey framework uses detailed mask regions to implement object-based representations. The Osprey model employs a mask aware visual extractor component to capture pixel-level features inside each object region. The mask ware visual extractor component encodes mask-level visual features, and moreover, gathers the spatial position information of every region. 

To implement this, Osprey first uses the multi-level image features generated by the vision encoder to adopt the mask-pooling operation, and for each single-;evel feature, the framework pools all of the features that lie throughout the mask region. The model then encodes the features across different layers by passing each feature through a linear projection layer that generates region-level embeddings, and fuses multi-level features by performing summation. The model then uses a MLP layer to provide the visual mask token. Moreover, Osprey preserves the spatial geometry of the thing region by encoding the pixel-level position relationship by implementing a binary mask for every object region. Ultimately, Osprey includes the visual mask token and its respective spatial tokens for every mask region embedding. 

LLM Tokenization

As mentioned earlier, the model extracts the image-level embeddings of a picture by feeding it right into a pre-trained CNN-based visual encoder. For textual information, the model first uses pre-trained LLM tokenizers to tokenize text sequences, after which projects these tokenized text sequences into text embeddings. For mask-based regions, the model defines a special token as a placeholder, after which substitutes it with a spatial token together with a mask token. When the model refers to an object region within the text input, it appends the placeholder after its region name that enables the mask regions to combine with texts well leading to complete sentences without the tokenization space. Furthemore, other than user instructions, the model also features a prefix prompt, a special token that serves as a placeholder, that’s then replaced by the vision encoder’s image-level embeddings. Finally, the framework interleaves the region-level & image-level visual tokens together with text tokens, and feeds it into the big language model to grasp the user instructions and the image with different regions in the thing. 

Osprey : Three Stage Training Process

The Osprey framework deploys a 3 stage training process by which each of the training phases is supervised by minimizing a next-token prediction loss.

Stage 1: Image-Text Alignment Training

In the primary stage, the Osprey framework deploys the CNN-based CLIP vision encoder to coach the image-level features and language connector to coach the model for image-text feature alignment. In the primary stage, the framework employs three components: a pre-trained large language model, a pre-trained vision encoder, and an image-level projector. The framework also adopts a MLP layer to serve because the vision-language connector that helps in enhancing Osprey’s multimodal generative capabilities. 

Stage 2: Mask-Text Alignment Pre-Training

Within the second stage, Osprey loads the load trained in the primary stage, and employs its Mask-Aware Visual Extractor component to capture pixel-level region features. Within the second stage, the framework only trains the Mask-Aware Visual Extractor to align language embeddings with mask-based region features. Moreover, the model collects pixel-level mask pairs and short texts from part-level and publicly-available object-level datasets, and converts them into instruction-following data to further train the model. 

Stage 3: End-to-End Nice Tuning

Within the third and the ultimate stage, the model fixes the weights of the vision encoder, and finetunes the big language model, mask-based region feature extractor, and the image-level projector components in its architecture. The first aim of coaching within the third stage is to increase the model’s capability to follow user instructions accurately, and efficiently perform pixel-level region understanding tasks. 

After implementing the three training stages, the Osprey framework is able to understanding complex scenarios defined by user instructions and based on pixel-level mask regions. 

Osprey : Experimental Results

To guage its performance, Osprey developers conduct a wide selection of experiments to show the model’s capabilities in classification, pixel-level region-based recognition, and sophisticated descriptions. 

Open-Vocabulary Segmentation

The first goal of open-vocabulary segmentation is to generate mask-based region recognition and its respective category explicitly. To attain open-vocabulary segmentation, Osprey first uses an input text prompt, following which the model adopts ground-truth mask regions for model interference to evaluate model’s performance in open-vocabulary recognition tasks. On the idea of the sentence response generated by the multimodal large language model, Osprey calculates the semantic similarity between the vocabulary list and output of every dataset. The next figure compares Osprey against cutting-edge multimodal large language models. 

As it may possibly be observed, the Osprey framework outperforms existing methods by a substantial margin on each the Cityscapes and the ADE20K-150 dataset. The outcomes indicate Osprey’s ability to outperform existing approaches, and achieve robust understanding and recognition on fine-grained object regions. 

Referring Object Classification

Within the Referring Object Classification task, the model is required to categorise the thing inside a selected region of a picture. To guage its classification capabilities, the Osprey framework uses two semantic relevance metrics including Semantic IoU or S-IoU and Semantic Similarity or SS. Semantic IoU represents the overlap of words between the ground-truth and the prediction labels whereas Semantic Similarity measures the similarity predicted and/or ground-truth labels in a semantic space. The next image demonstrates Osprey’s performance within the Referring Object Classification task when put against models employing box-level and image-level approaches. 

Detailed-Region Description

Within the Detailed-Region Description task, the model evaluates its performance on instruction-following detailed description capabilities together with other region-level approaches. The model randomly selects an input inference prompt from an inventory of predefined prompts, and leverages the GPT-4 LLM framework to measure the standard of the response generated by the model against the input referring regions comprehensively. Using the instruction generation pipeline, the model generates questions, and seeks GPT-4’s answers following which the LLM assesses the correctness of semantics and precision of referring understanding. The next table demonstrates the performance of Osprey against cutting-edge models on Detailed-Region Description tasks. 

Region-Level Captioning

The Osprey framework also outperforms current approaches on Region-Level Captioning tasks with the outcomes contained in the next image. 

Final Thoughts

In this text, we have now talked about Osprey, a mask-text instruction training method with the first aim of extending MLLMs by incorporating fine-grained masked regions in language instruction to realize pixel-level visual-language understanding. To perform its goal, the Osprey framework curates a mask-based region-text dataset with over 700 thousand samples, and injects pixel-level representation into LLM to design a vision-language model. The Osprey framework goals to boost MLLMs for fine-grained visual understanding significantly, and by implementing a CNN-based CLIP model and a mask-aware visual extractor, Osprey attains the aptitude to know images at each part-level and object-level regions.