A future version of DLSS is likely to include full neural rendering, hinted Bryan Catanzaro, a Nvidia VP of Applied Deep Learning Research. In a round table discussion organized by Digital Foundry (video), various video game industry experts talked about the future of AI in the business. During the discussion, Nvidia’s Catanzaro raised a few eyebrows with his openness to predict some key features of a hypothetical "DLSS 10."
We've seen significant developments in Nvidia’s DLSS technology over the years. First launched with the RTX 20-series GPUs, many wondered about the true value of technologies like the Tensor cores being included in gaming GPUs. The first ray tracing games, and the first version of DLSS, were of questionable merit. However, DLSS 2.X improved the tech and made it more useful, leading to it being more widely utilized — and copied, first via FSR2 and later with XeSS.
DLSS 3 debuted with the RTX 40-series graphics cards, adding Frame Generation technology. With 4x upscaling and frame generation, neural rendering potentially allows a game to only fully render 1/8 (12.5%) of the pixels. Most recently, DLSS 3.5 offered improved denoising algorithms for ray tracing games with the introduction of Ray Reconstruction technology.
The above timeline raises questions about where Nvidia might go next with future versions of DLSS. And of course, "Deep Learning Super Sampling" no longer really applies, as the last two additions have targeted other aspects of rendering. Digital Foundry asked that question to the group: “Where do you see DLSS in the future? What other problem areas could machine learning tackle in a good way?”
Bryan Catanzaro immediately brought up the topic of full neural rendering. This idea isn’t quite as far out as it may seem. Catanzaro reminded the panel that, at the NeurIPS conference in 2018, Nvidia researchers showed an open-world demo of a world being rendered in real-time using a neural network. During that demo the UE4 game engine provided data about what objects were in a scene, where they were, and so on, and the neural rendering provided all the on-screen graphics.
The graphics back in 2018 were rather basic — “Not anything close to Cyberpunk,” admitted Catanzaro. Advances in AI image generation have been quite incredible in the interim, though. Look at the leaps in quality we've seen in AI image generators over the last year, for example.
Catanzaro suggested the 2018 demo was a peek into a major growth area of (generative) AI in gaming. “DLSS 10 (in the far far future) is going to be a completely neural rendering system,” he speculated. The result will be “more immersive and more beautiful” games than most can imagine today.
Between now and DLSS 10, Catanzaro believes we'll see a gradual process, developer controllable, and coherent. Developers are now experienced in tools that allow them to steer their vision using traditional game engines and 2D / 3D rendering tech. They need similar finely controlled tools ready for generative AI, noted the Nvidia VP.
Another interesting perspective on the future of AI in gaming was provided by Jakub Knapik, VP Art and Global Art Director at CD Projekt RED. Knapik was fully transparent in saying Catanzaro’s easily believable vision of generative AI graphics “scares me like crazy.” However, he sounded more excited about AI enhancing the interactive and immersive worlds that make gaming so much fun, not just graphics.
Currently, 90% of a game might be prescriptive, noted the senior CDPR developer. In contrast, he referenced Nvidia’s ‘Cyberpunky’ demo of an AI-powered interactive NPC as providing a window into deeper AI-enhanced games. Knapik went on to note that for CDPR, the story is king, and that’s where he sees the best use of AI going forward.
Both Catanzaro and Knapik seemed to agree that developers need extensive control over AI to realize their creative visions. People won’t just be able to conjure up an AAA game from a chat prompt, joked the Nvidia VP. We can see it now: "/Imagine: A dystopian future first-person game with RPG elements in a violent world dominated by corporations."
Wherever we go in the interim, by the time DLSS 10 rolls out, we could see game engines wielding AI-generated visuals for better real-time graphics quality and performance, with AI-enhanced characters, stories, and environments contributing to what might become a whole new category of media.
