In recent times, Diffusion Models (DMs) have made significant strides within the realm of image synthesis. This has led to a heightened concentrate on generating photorealistic images from text descriptions (T2I). Constructing upon the accomplishments of T2I models, there was a growing interest amongst researchers in extending these techniques to the synthesis of videos controlled by text inputs (T2V). This expansion is driven by the anticipated applications of T2V models in domains corresponding to filmmaking, video games, and artistic creation.

Achieving the proper balance between video quality, training cost, and model compositionality stays a posh task, necessitating careful considerations in model architecture, training strategies, and the gathering of high-quality text-video datasets.

In response to those challenges, a brand new integrated video generation framework called LaVie has been introduced. This framework, boasting a complete of three billion parameters, operates using cascaded video latent diffusion models. LaVie serves as a foundational text-to-video model built upon a pre-trained T2I model (specifically, Stable Diffusion, as presented by Rombach et al., 2022). Its primary goal is to synthesize visually realistic and temporally coherent videos while retaining the creative generation capabilities of the pre-trained T2I model.

Figure 1 above demonstrates Text-to-video samples and Figure 2 demonstrates Diverse video generation results by Lavie. 

LaVie incorporates two key insights into its design. First, it utilizes easy temporal self-attention coupled with RoPE to effectively capture inherent temporal correlations in video data. Complex architectural modifications provide only marginal improvements within the generated results. Second, LaVie employs joint image-video fine-tuning, which is important for producing high-quality and inventive outcomes. Attempting to fine-tune directly on video datasets can compromise the model’s ability to combine concepts and result in catastrophic forgetting. Joint image-video fine-tuning facilitates large-scale knowledge transfer from images to videos, encompassing scenes, styles, and characters.

Moreover, the publicly available text-video dataset, WebVid10M, is found to be inadequate for supporting the T2V task on account of its low resolution and concentrate on watermark-centered videos. In response, LaVie advantages from a newly introduced text-video dataset named Vimeo25M, which comprises 25 million high-resolution videos (> 720p) accompanied by text descriptions. 

Experiments display that training on Vimeo25M significantly enhances LaVie’s performance, allowing it to generate superior results by way of quality, diversity, and aesthetic appeal. Researchers envision LaVie as an initial step towards achieving high-quality T2V generation. Future research directions involve expanding the capabilities of LaVie to synthesize longer videos with intricate transitions and movie-level quality based on script descriptions.

Try the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 31k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.

If you happen to like our work, you’ll love our newsletter..