Adobe integrates Firefly Video and third-party AI models into Premiere enabling editors to add AI-generated content to films

Adobe has integrated generative AI into Adobe Premiere to allow editors to quickly add artificial intelligence created content into films. The AI models include Adobe Firefly Video as well as third party models such as Pika and Sora. Adobe plans to allow further third-party integration by creating an AI plug-in standard.

Generative artificial intelligence has advanced rapidly, and today, AI models such as Sora can quickly create convincing, short video clips that include realistic actors and scenes based on text prompts. While the technology to create a blockbuster film from a script does not exist, current AI technology is useful enough to assist editors.

Adobe has demonstrated three major ways that AI can help, and readers who want to jump into Adobe Premiere can buy a subscription today along with the official training book.

The first is the creation of new B-roll to help transition between scenes or fill in for missing shots. While large film productions closely follow a detailed shot list, smaller productions may not, so a critical shot might be missed. Instead of spending more money to go back to the location to reshoot a missing shot, AI such as Sora can create new shots from text prompts. As the technology advances, the ability of AI to create well-matched shots will continue to improve.

The second is the addition and deletion of objects from scenes. Traditionally, special effects artists would need to manually mask, track, and add or remove objects. This can take hours for a simple object such as an unwanted sign or animal because the replacement background takes much time to create in a 3D editing application such as Maya or paint in a 2D application like Photoshop. Adobe Firefly Video simplifies this by leveraging the speed of AI to automatically add or replace objects while seamlessly replacing gaps in the background.

The third is the simpler task of extending a video by a few frames or seconds. Typically, film shoots hold on a scene longer than written by a few seconds so that editors can cut between scenes cleanly. But in some cases, a scene is too short, so the transition between two scenes becomes awkward because it is too quick. Firefly AI can create brand-new frames to extend the clip in a realistic, seamless manner.

One potential issue of Adobe’s AI integration approach is that they intend to tag all AI-generated content with Content Credentials with information regarding how it was made and the AI models used. This opens privacy concerns since an artist’s trade secrets can be exposed, potentially along with personal or proprietary information.