Text-to-image models have recently gained numerous attention. With the introduction of Generative Artificial Intelligence, models like GPT and DALL-E have been within the headlines ever since their release. Their rise in popularity is the rationale due to why generating content like a human is not any longer a dream today. Not only text-to-image models but additionally text-to-video (T2V) generation is now possible. Filming live-action or producing computer-generated animation is usually required to provide interesting storytelling videos, which is a difficult and time-consuming procedure. 

Though the newest advancements in text-to-video production have demonstrated promise in mechanically creating videos from text-based descriptions, there are still certain limitations. Lack of control over the resulting video’s design and layout, that are essential for visualizing an interesting story and producing a cinematic experience, is a primary challenge. Close-ups, long views, and composition, amongst other filmmaking techniques, are crucial in allowing the audience to know subliminal messages. Currently, existing text-to-video methods struggle to supply appropriate motions and layouts that adhere to the standards of cinema.

To deal with the constraints, a team of researchers has proposed a singular video generation approach, which is retrieval-augmented video generation, called Animate-A-Story. This method takes advantage of the abundance of existing video content by obtaining movies from external databases based on text prompts and using them as a guide signal for the T2V creation process. Users can have greater control over the layout and composition of the generated videos when animating a story, using the input retrieved videos as a structure reference.

The framework consists of two modules: Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. The Motion Structure Retrieval module supplies video candidates that match the requested scene or motion context as indicated by query texts. For this, video depths are extracted as motion structures using a industrial video retrieval system. The second module, Structure-Guided Text-to-Video Synthesis, uses the text prompts and motion structure as input to provide movies that follow the storyline. A model has been created for customizable video production that permits flexible control over the plot and characters of the video. The created videos adhere to the intended storytelling elements by following the structural direction and visual guidelines.

This approach places a powerful emphasis on preserving visual coherence between footage. The team has also developed a successful concept personalization technique to ensure this. Through text prompts, this method enables viewers to pick preferred character identities, preserving the uniformity of the characters’ appearances throughout the video. For evaluation, the team has compared the approach to existing baselines. The outcomes demonstrated significant benefits of this approach, proving its capability to generate high-quality, coherent, and visually engaging storytelling videos. 

The team has summarized the contribution as follows: 

1. A retrieval-augmented paradigm for narrative video synthesis has been introduced, which, for the primary time, allows the use of assorted existing videos for storytelling.

2. The framework’s usefulness is supported by experimental findings, which establish it as a cutting-edge tool for creating videos which can be remarkably user-friendly.

3. A versatile structure-guided text-to-video approach has been proposed that successfully reconciles the stress between character production and structure guiding.

4. The team has also introduced TimeInv, a brand new concept within the personalization approach that significantly exceeds its current rivals.

Try the Paper, Github, and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to affix our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

🚀 Check Out 900+ AI Tools in AI Tools Club