Home Community This AI Paper from UC Berkeley Advances Machine Learning by Integrating Language and Video for Unprecedented World Understanding with Modern Neural Networks

This AI Paper from UC Berkeley Advances Machine Learning by Integrating Language and Video for Unprecedented World Understanding with Modern Neural Networks

0
This AI Paper from UC Berkeley Advances Machine Learning by Integrating Language and Video for Unprecedented World Understanding with Modern Neural Networks

Current approaches to world modeling largely give attention to short sequences of language, images, or video clips. This implies models miss out on information present in longer sequences. Videos encode sequential context that may’t be easily gleaned from text or static images. Long-form text holds information unobtainable briefly pieces and is vital to applications like document retrieval or coding. Processing long video and text sequences together could enable a model to develop a broader multimodal understanding, making it a potentially powerful tool for a lot of tasks.

Directly modeling tens of millions of tokens is amazingly difficult resulting from the high computational cost, memory constraints, and lack of suitable datasets. Fortunately, RingAttention allows us to scale to longer context sizes without overheads, enabling efficient training on long sequences.

Researchers need a big dataset of long videos and language sequences to harness this capability. They’ve curated this dataset from publicly available books and video sources containing videos of diverse activities and books on various topics. To scale back training costs, they’ve regularly increased context size from 4K to 1M tokens to scale back this cost, and this approach performs well in extending context effectively.

Researchers face several challenges when training on video and language concurrently. They found that combining video, images, and text is vital to balancing visual quality, sequential information, and linguistic understanding. They achieve this by implementing an efficient type of masked sequence packing for training with different sequence lengths. Moreover, determining the proper balance between image, video, and text training is crucial for cross-modal understanding, and the researchers suggest an efficient ratio. Moreover, to take care of the dearth of long-form chat datasets, they developed a technique where a short-context model is used to generate a question-answering (QA) dataset from books, which proves crucial for long-sequence chat ability.

Let’s discuss the general method intimately; the researchers train a big autoregressive transformer model on a large dataset, incrementally increasing its context window to 1,000,000 tokens. They construct on Llama2 7B and mix long multimodal sequences with text-image, text-video data, and books. The training stages and datasets are shown in Figure 3, and the model architecture is shown in Figure 4.

Training Stages and Datasets

  • Stage I: Long-Context Language Model
    • Extending Context: The researchers use RingAttention for scalable long-document training and adjust positional encoding parameters.
    • Progressive Training: They save compute by training on increasingly longer sequences (32K to 1M tokens).
    • Chat High-quality-tuning: They generate a QA dataset for long-context chat abilities from books.
  • Stage II: Long-Context Vision-Language Models
    • Architectural Changes: The researchers use VQGAN tokens for images and videos and add markers to modify between vision and text generation.
    • Progressive Training: The model undergoes several stages of coaching, each increasing in sequence length. This step-by-step approach helps learn more effectively by starting with simpler tasks before moving on to more complex sequences.
    • Chat Datasets: They include various types of chat data for his or her goal downstream tasks. By generating a question-answering dataset from long texts, the model improves its ability to have interaction in meaningful conversations over prolonged sequences.
https://arxiv.org/abs/2402.08268

On evaluation, the model achieves near-perfect retrieval accuracy over its entire 1M context window and scales higher than current LLMs. It also performs competitively in multi-needle retrieval and exhibits strong short-context language performance (shown in Figure 2), indicating successful context expansion. While it has some limitations, resembling difficulty with complex long-range tasks, it provides a foundation for future work, including developing more difficult benchmarks.

In conclusion, this pioneering work has set a brand new benchmark in AI’s capability to understand the world by integrating language and video. Leveraging the revolutionary RingAttention mechanism, the study demonstrates scalable training on an in depth dataset of long videos and books, progressively expanding the context size from 32K to an unprecedented 1M tokens. This approach, combined with the event of masked sequence packing and loss weighting techniques, enables the efficient handling of a various array of content. The result’s a model with a 1M token context size, the most important so far, adept at navigating the complexities of lengthy video and language sequences. With the open sourcing of this optimized implementation and a 7B parameter model, the research invites further innovation in the sphere, aiming to boost AI’s reasoning abilities and understanding of the world.

Nonetheless, the journey doesn’t end here. Despite its significant achievements, the work acknowledges limitations and areas ripe for future exploration. Enhancing video tokenization for more compact processing, incorporating additional modalities like audio, and improving video data quality and quantity are critical next steps. These advancements promise to further refine AI’s multimodal understanding, opening recent pathways for research and application in the search to develop more sophisticated and capable AI systems.


Take a look at the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 38k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

Should you like our work, you’ll love our newsletter..

Don’t Forget to hitch our Telegram Channel

You could also like our FREE AI Courses….


Vineet Kumar is a consulting intern at MarktechPost. He’s currently pursuing his BS from the Indian Institute of Technology(IIT), Kanpur. He’s a Machine Learning enthusiast. He’s captivated with research and the most recent advancements in Deep Learning, Computer Vision, and related fields.


🚀 LLMWare Launches SLIMs: Small Specialized Function-Calling Models for Multi-Step Automation [Check out all the models]

LEAVE A REPLY

Please enter your comment!
Please enter your name here