Google researchers unveil Genie – an AI that knows how to create mini-game scenes from example images

Researchers from Google DeepMind have unveiled Genie, the world’s first generative interactive environment AI. The AI can create a playable scene after looking at an image of an example game. This opens the door for gamers to create playable games without programming.

Genie is trained on 2D platformer gameplay by looking at 30,000 hours of videos (6.8 million 16 second clips). The AI simplifies what it sees by representing the video images and the actions it sees as sets of numbers, much like a human using words to describe a game scene.

Once trained, a single example of a game scene is all Genie needs to create a playable mini-scene. Behind the scenes, the AI can extend the starting image using sophisticated methods for generating missing parts of an image, so Genie can draw new parts of the game world as the player moves forward. It does this by referring to the millions of clips it has seen while giving the character playable actions such as running or jumping.

Importantly, Genie learns actions by watching gameplay, not by having humans point this out. So when there’s a bridge with a gap, the AI knows from what it has seen that a character will likely need to jump over the gap and does so when the human player presses the up button on a controller.

This all takes enormous computing power for training (thousands of teraflops over weeks on computers with hundreds of chips) and gameplay, which it can only do at a super-slow 1 frame per second.

For the technically minded, video frames and inferred latent actions are tokenized utilizing using van den Oord’s VQ-VAE. User-specified latent actions along with tokenized video frames are fed into the dynamics model utilizing Chang’s MaskGIT transformer to generate iterative frame tokens that are used to generate output videos utilizing Gupta’s MaskViT. ST-transformers adopted from Xu are used throughout to include temporal attention in addition to spatial to improve predictions as well as reduce memory requirements.

So, until the day arrives when computers are powerful enough to create 2D platformer games completely on their own, enjoy classic platformers with a handheld game console (like this at Amazon).