Notebookcheck

Facebook researchers develop AI method to convert videos of people into playable game characters

This new AI method of converting video to playable characters may be the evolution of the Animoji and its ilk. (Source: MacRumors)
This new AI method of converting video to playable characters may be the evolution of the Animoji and its ilk. (Source: MacRumors)
A trio of IT researchers affiliated with Facebook have created a new method to turn a video of an individual performing a series of actions (e.g. a dance) into a 3D 'game character'. This character - and its associated movements - can then be transposed onto a new background. The researchers assert that this background could interact with said character.

Apple recently unveiled the power of AI in mapping facial data so as to convert it into an animated character. This innovation, known as the Animoji, may realize the dreams of many to turn themselves into their favorite avatars as much as possible on a digital platform. The next steps in this use of technology may be to also turn oneself into one's own game character. A group of Facebook researchers have published a new paper that suggests this possibility is closer than ever.

These scientists - Oran Gafni, Lior Wolf and Yaniv Taigman - have developed a system by which a person and their movements are captured on video, then converted into a 3D character that can replicate the motions or gestures in question in response to joystick-based controls. This sounds like an impressive breakthrough; however, as the researchers concede themselves in their article, this has actually been done (or at least started on) in prior projects.

As in this body of work, Gafni, Wolf & Taigman used AI in order to model, mask and eventually produce their video-to-playable-characters. The researchers used 2 such programs, Pose2Pose and Pose2Frame, which 'learned' the subjects' body shapes, movement styles and other characteristics in order to build up the resulting characters.

However, the authors also assert that their new method improves on earlier counterparts, in that their masking and output techniques result in characters that appear more realistic against their backgrounds. In other words, they relate better to their surroundings; for example, the shadows they cast are more realistic and accurate.

This also means that the Facebook scientists' characters can be imposed onto different backgrounds with greater ease and efficacy than before. Their movements are also convincing enough, although they could be a little more natural. Nevertheless, this remains a potentially significant advancement in the field of AI-enhanced media conversions. It will be interesting to see where this research - which is available in Cornell's arXiv database of advance research publications - takes us in the future.

Working For Notebookcheck

Are you a techie who knows how to write? Then join our Team! Especially English native speakers welcome!

Currently wanted: 
News and Editorial Editor - Details here

Source(s)

static version load dynamic
Loading Comments
Comment on this article
Please share our article, every link counts!
> Notebook / Laptop Reviews and News > News > News Archive > Newsarchive 2019 04 > Facebook researchers develop AI method to convert videos of people into playable game characters
Deirdre O Donnell, 2019-04-22 (Update: 2019-04-22)