Nvidia shows off interactive NPCs and DLSS 3.5 ray reconstruction coming to all RTX GPUs this fall

Apart from announcing a Half-Life 2 RTX remaster at Gamescom 2023, Nvidia also showed off some improvements coming to its RTX technologies. These include a large language model (LLM) for in-game non-playable characters (NPCs) and a major DLSS 3.5 update that offers better lighting effects via Ray Reconstruction.

Nvidia ACE and NeMo SteerLM

NPCs add to the immersion in a game world, but it is a herculean task for developers to distinctly program all NPCs in a game. Nvidia first showed off its Avatar Cloud Engine (ACE) back during Computex 2023 that leverages AI to bring intelligence to NPCs.

ACE relies on three main AI-powered components — NeMo LLM, Riva automatic speech recognition (ASR) and text-to-speech (TTS) capabilities, and Omniverse Audio2Face that can create facial animation from an audio source.

Now, the company is augmenting ACE with NeMo SteerLM, which is an LLM that adds personality and emotion to NPCs for added realism. SteerLM allows gamers to define and fine-tune character interaction in a manner similar to how LLMs can be guided using natural language prompts. SteerLM also allows developers to allow models to be influenced by events in the game world.

DLSS 3.5 brings Ray Reconstruction to all RTX GPUs

Nvidia introduced DLSS 3 with Frame Generation that leverages the 4th gen Tensor cores and Optical Flow Accelerators in RTX 40 Series GPUs. Now, at Gamescom 2023, Nvidia is now bringing the ability to include ray tracing pipelines with DLSS 3.5.

A traditional ray tracing pipeline samples rays for reflections and global illumination which is then subjected to denoising. In DLSS, this is performed at a lower resolution and then upscaled. However, the use of denoisers, though essential, can result in inaccurate lighting while also potentially removing information, such as color.

Currently, developers employ the Nvidia Real-time Denoiser (NRD), which is a big improvement over the Spatiotemporal Variance-Guided Filtering (SVGF) solution used earlier. Rumors indicate that Nvidia will include a denoising accelerator directly in the RT pipeline with the upcoming RTX 50 Blackwell cards.

That is just speculation though, and RTX 50 is still a couple of years away from seeing the light of the day. For now, Nvidia aims to overcome denoising limitations with DLSS 3.5, which brings forth AI-powered Ray Reconstruction (RR).

The company said RR is trained on 5x more data than DLSS 3 and is a superior solution that is trained to recognize motion vectors and ray traced effects while being able to differentiate between good and pixel pixels. The AI network is also trained to retain vital information to generate a high-quality upscale.

According to Nvidia, toggling DLSS 3.5 should have no major hit on overall game performance. Unlike DLSS 3.0 Frame Generation (FG) that explicitly makes use of Optical Flow Accelerators in RTX 40 Series GPUs, DLSS 3.5 RR will work on all RTX cards.

DLSS 3.5 will be coming this fall to games such as Cyberpunk 2077: Phantom Liberty, Portal with RTX, and Alan Wake 2 and will be supported on platforms such as Omniverse, Chaos Vantage, and D5 Render.