Lately, there was a rapid development in text-based visual content generation. Trained with large-scale image-text pairs, current Text-to-Image (T2I) diffusion models have demonstrated a formidable ability to generate high-quality images based on user-provided text prompts. Success in image generation has also been prolonged to video generation. Some methods leverage T2I models to generate videos in a one-shot or zero-shot manner, while videos generated from these models are still inconsistent or lack variety. Scaling up video data, Text-to-Video (T2V) diffusion models can create consistent videos with text prompts. Nevertheless, these models generate videos lacking control over the generated content. 

A recent study proposes a T2V diffusion model that permits for depth maps as control. Nevertheless, a large-scale dataset is required to attain consistency and top quality, which is resource-unfriendly. Moreover, it’s still difficult for T2V diffusion models to generate videos of consistency, arbitrary length, and variety.

Video-ControlNet, a controllable T2V model, has been introduced to handle these issues. Video-ControlNet offers the next benefits: improved consistency through the usage of motion priors and control maps, the power to generate videos of arbitrary length by employing a first-frame conditioning strategy, domain generalization by transferring knowledge from images to videos, and resource efficiency with faster convergence using a limited batch size.

Video-ControlNet’s architecture is reported below.

The goal is to generate videos based on text and reference control maps. Subsequently, the generative model is developed by reorganizing a pre-trained controllable T2I model, incorporating additional trainable temporal layers, and presenting a spatial-temporal self-attention mechanism that facilitates fine-grained interactions between frames. This approach allows for the creation of content-consistent videos, even without extensive training.

To make sure video structure consistency, the authors propose a pioneering approach that comes with the motion prior of the source video into the denoising process on the noise initialization stage. By leveraging motion prior and control maps, Video-ControlNet is in a position to produce videos which can be less flickering and closely resemble motion changes within the input video while also avoiding error propagation in other motion-based methods as a consequence of the character of the multi-step denoising process.

Moreover, as an alternative of previous methods that train models to directly generate entire videos, an revolutionary training scheme is introduced on this work, which produces videos predicated on the initial frame. With such a simple yet effective strategy, it becomes more manageable to disentangle content and temporal learning, as the previous is presented in the primary frame and the text prompt. 

The model only must learn generate subsequent frames, inheriting generative capabilities from the image domain and easing the demand for video data. During inference, the primary frame is generated conditioned on the control map of the primary frame and a text prompt. Then, subsequent frames are generated, conditioned on the primary frame, text, and subsequent control maps. Meanwhile, one other advantage of such a method is that the model can auto-regressively generate an infinity-long video by treating the last frame of the previous iteration because the initial frame.

That is how it really works. Allow us to take a have a look at the outcomes reported by the authors. A limited batch of sample outcomes and comparison with state-of-the-art approaches is shown within the figure below.

This was the summary of Video-ControlNet, a novel diffusion model for T2V generation with state-of-the-art quality and temporal consistency. In the event you have an interest, you possibly can learn more about this method within the links below.

Check Out The Paper. Don’t forget to hitch our 25k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you might have any questions regarding the above article or if we missed anything, be happy to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club