Notebookcheck Logo

Meta unveils Movie Gen AI to create videos and music using text prompts to help filmmakers save time and money

Meta unveils Movie Gen AI to create videos and music using text prompts to help filmmakers save time and money. (Image source: Meta)
Meta unveils Movie Gen AI to create videos and music using text prompts to help filmmakers save time and money. (Image source: Meta)
Meta has unveiled Movie Gen, an AI that can create and edit videos with music and sound effects using text prompts. Meta researchers suggest its AI is state-of-the-art in its video and audio generation capabilities. The AI can animate a person from one photograph, replace any object with another, and create professional soundtracks and sound effects.
AI Software

Meta has unveiled Movie Gen, AI software that only needs typed text prompts to create and edit realistic-looking videos with professional-sounding music and sound effects.

Movie Gen is a 30-billion parameter AI model capable of generating 16-second HD clips from text prompts. The AI has been pre-trained on one billion images and 100 million videos. The videos have been culled from a much larger collection for quality to improve the training. Movie Gen Audio is a 13-billion parameter AI model capable of generating 48 kHz sound effects and music from text prompts. The AI has been pre-trained on one million hours of audio. Human feedback and high-quality audio and video examples were used to fine tune the AI.

Given a photograph of a person and a description of that person in a scene, Movie Gen can generate realistic video of an animated actor and scene. The AI has been trained on 22 camera motion and position types, such as wide angle, tilt up, and truck left, allowing filmmakers to specify virtual camera placement and moves like a real shoot. Meta researchers reported that Movie Gen has the ability to edit videos with high realism and high accuracy in following the text prompts.

Users can also use Movie Gen Audio to add professional-sounding audio to the clips, including sound effects and music scores based on text prompts. The AI can create music several minutes long, but is limited to 16-second video clips due to the enormous computer resources this demands. Meta researchers reported that Movie Gen Audio has the ability to create realistic on-screen and off-screen sounds such as footsteps and music that are well-timed to on-screen action and supports the emotional arc of a video.

Meta researchers suggest that their AI is state-of-the-art in its video and audio generation capabilities in their research paper. They came to this conclusion after internal testing using human evaluators of Movie Gen features against the top, competing generative audio and video tools available. Generative AI software is a rapidly developing field, so the reported capabilities of AI in development and the actual quality of generated output will differ from that of released software. Meta is working on adding safeguards to Movie Gen and will release the AI once it feels confident in its safety.

Readers who don't mind AI replacing human actors and film crews can enter the AI era with an extremely fast laptop like the Asus ROG Strix Scar 17 X3D (here on Amazon) to run generative, 3D modeling, and photo editing software with ease. Filmmakers who insist on making films with real actors and sets can use a top-notch DSLR (like the latest Nikon Z6III on Amazon) on their IRL shoots to showcase their AI-beating talents.

The ability of generative AI to create realistic images that follow text prompts accurately is the basis for high-quality generative video. (Image source: Meta)
The ability of generative AI to create realistic images that follow text prompts accurately is the basis for high-quality generative video. (Image source: Meta)
Meta researchers found the text-to-image capabilities of Movie Gen to be among the best available at the time under ELO scoring of images using human evaluators. (Image source: Meta)
Meta researchers found the text-to-image capabilities of Movie Gen to be among the best available at the time under ELO scoring of images using human evaluators. (Image source: Meta)
Meta Movie Gen can create sound effects and music to match any scene from a text prompt. Listen to examples in the playlist linked below. (Image source: Meta)
Meta Movie Gen can create sound effects and music to match any scene from a text prompt. Listen to examples in the playlist linked below. (Image source: Meta)
Human evaluators listening to Meta Movie Gen Audio text-to-audio creations rated them to be better sounding and better aligned with the video than the output from competitors. (Image source: Meta)
Human evaluators listening to Meta Movie Gen Audio text-to-audio creations rated them to be better sounding and better aligned with the video than the output from competitors. (Image source: Meta)
Meta Movie Gen has the ability to create and light sets, animate actors from a single photograph, and add special effects like smoke without the real-life set costs and hassles. (Image source: Meta)
Meta Movie Gen has the ability to create and light sets, animate actors from a single photograph, and add special effects like smoke without the real-life set costs and hassles. (Image source: Meta)
Human evaluators found Meta Movie Gen text-to-video output to be higher-quality, more realistic, and more natural than those of competitors. (Image source: Meta)
Human evaluators found Meta Movie Gen text-to-video output to be higher-quality, more realistic, and more natural than those of competitors. (Image source: Meta)
One photo is all Meta Movie Gen needs to animate actors across an infinite number of AI-generated scenes. (Image source: Meta)
One photo is all Meta Movie Gen needs to animate actors across an infinite number of AI-generated scenes. (Image source: Meta)
Meta Movie Gen learns video editing by making an edit to a still image while generating a clip from a text prompt. Next, the AI identifies the object to be edited and practices creating a realistic, edited video. (Image source: Meta)
Meta Movie Gen learns video editing by making an edit to a still image while generating a clip from a text prompt. Next, the AI identifies the object to be edited and practices creating a realistic, edited video. (Image source: Meta)
Meta researchers found Meta Movie Edit to be superior to competing AI in creating high-quality, realistic edits that follow text prompts accurately. (Image source: Meta)
Meta researchers found Meta Movie Edit to be superior to competing AI in creating high-quality, realistic edits that follow text prompts accurately. (Image source: Meta)
Read all 3 comments / answer
static version load dynamic
Loading Comments
Comment on this article
Cybertruck owner loses GEICO insura...
Qualcomm Snapdragon 8 Gen 4 specs r...
Please share our article, every link counts!
Mail Logo
> Expert Reviews and News on Laptops, Smartphones and Tech Innovations > News > News Archive > Newsarchive 2024 10 > Meta unveils Movie Gen AI to create videos and music using text prompts to help filmmakers save time and money
David Chien, 2024-10- 7 (Update: 2024-10- 7)