Powering the meteoric rise of AI chatbots, LLMs are the talk of the town. They’re showing mind-blowing capabilities in user-tailored natural language processing functions but appear to be lacking the flexibility to grasp the visual world. To bridge the gap between the vision and language world, researchers have presented the All-Seeing (AS) project. 

The AS Project is for open-world panoptic visual recognition and understanding, driven by the goal of making a vision system that mimics human cognition. The term “panoptic” refers to including the whole lot visible in a single view.

The AS Project consists of:

• The All-Seeing 1B (AS-1B) dataset covers a big selection of three.5 million common and rare concepts in the actual world and has 132.2 billion tokens that describe the concepts and their attributes.
• The All-Seeing model (ASM) is a unified location-aware image-text foundation model. The model consists of two key components: a location-aware image tokenizer and an LLM-based decoder.

The dataset comprises over 1 billion region annotations in various formats, comparable to semantic tags, locations, question-answering pairs, and captions. Compared with the previous visual recognition datasets like ImageNet and COCO, visual understanding datasets like Visual Genome and Laion-5B, the AS-1B dataset stands out resulting from its wealthy and diverse instance-level location annotation and corresponding detailed object concepts and descriptions. 

The architecture of the AS model comprises a unified framework of various levels. It supports contrastive and generative image-text tasks at each the image level and region levels. By leveraging pre-trained LLMs and powerful vision foundation models (VFMs), the model demonstrates promising performance in discriminative tasks like image-text retrieval and nil classification, in addition to generative tasks comparable to visual query answering (VQA), visual reasoning, image captioning, region captioning/VQA, etc. Moreover, researchers claim to see potential in grounding tasks like phrase grounding and referring expression comprehension with the help of a class-agnostic detector.

The All-Seeing Model (ASM) comprises of three key designs:

1. A location-aware image tokenizer extracts feature from the image and region levels based on the input image and bounding box, respectively.
2. A trainable task prompt is incorporated at the start of the vision and text tokens to guide the model in distinguishing between discriminative and generative tasks.
3. An LLM-based decoder is utilized to extract vision and text features for discriminative tasks and auto-regressively generate response tokens in generative tasks. 

Extensive data evaluation by way of quality, scaling, diversity, and experiments was conducted by analyzing and comparing the proposed ASM with a CLIP-based baseline model(displays zero-shot capabilities of GPT-2 and three) and leading Multimodality Large Language models (VLLMs) on representative vision tasks including zero-shot region recognition, image-level caption, and region-level caption. The findings highlighted the strong region-level text generation capabilities of our model while also showcasing its ability to understand the complete image. The human evaluation results indicated that captions generated by our ASM are preferred over those from MiniGPT4 and LLaVA.

The model is trained with open-ended language prompts and locations, which allows it to generalize to numerous vision and language tasks with remarkable zero-shot performance, including region-text retrieval, region recognition, captioning, and question-answering. This, in keeping with researchers, has given LLMs a “all-seeing eye” and has revolutionized the intersection of vision and language.

Take a look at the Paper and GitHub. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.