Unveiling the Secrets Behind the PlayStation 5: A Candid Conversation with Mark Cerny and Mike Fitzgerald

Sony’s PlayStation 5 Pro is a powerful mid-generation enhancement that provides machine learning upscaling, enhanced ray tracing, and more raw graphical power. We’ve invested considerable time testing and exploring its features, yet it’s undeniably intriguing to consider the other side: how it was constructed and the design philosophy behind its software. With this in mind, Digital Foundry’s Oliver Mackenzie interviewed PS5 Pro lead system architect Mark Cerny and core technology director at Insomniac Games, Mike Fitzgerald.

This technical conversation spans a variety of topics during its 30-minute duration, including the challenges and opportunities presented by PSSR upscaling, the reasoning behind the new console’s hardware design, and how developers have approached the new possibilities—both in strategy and performance—that are now available. This segment accompanies a comprehensive analysis of the PS5 Pro’s architecture made during the interview, which took place on October 30th, so while the subjects discussed are broad, details such as the varying implementations of PSSR across different games weren’t detailed at the time of the conversation.

The dialogue also addresses where Sony is looking to progress in the future. As Rich suggested in his PS5 Pro review, in addition to being a robust machine right now, this console offers a significant opportunity for Sony and game developers alike to familiarize themselves with the RT and ML technology that will shape the next generation of gaming—and this is reiterated in the conversation here. If you’re seeking to glean some insights regarding Sony’s direction with PlayStation 6, there’s a lot to reflect upon in this segment. As always, the text has been lightly edited for length and clarity.

Digital Foundry: I wanted to begin our discussion by talking about PSSR, PlayStation Spectral Super Resolution. There has been a noticeable difference in capability between developers like Naughty Dog and Square Enix regarding the options they’re providing. Some studios focus on delivering numerous modes, while others are using PSSR as a means to unify their various modes into a singular PS5 Pro experience. Do you have a preference for either of these methods?

Mark Cerny: My preference is that the game developers choose what they believe is best for their titles, so this is definitely a question for Mike.

Mike Fitzgerald: When we create a game and collaborate as a studio to deliver it, we make some really challenging trade-offs between what we want to present at a higher frame rate and what we can achieve at 30fps. The exciting aspect of working with new hardware is that suddenly, these compromises we made during the game’s development get left behind. We’re able to combine a lot of what defined the fidelity mode with what characterized the performance mode on the PS5 Pro. After spending some time aligning that and achieving a very solid performance mode, attention starts to shift towards considering that some players still prefer 30fps, and with this fantastic new hardware, what other intriguing features can we implement?

Digital Foundry: Yes, it indeed depends greatly on the game. For the Insomniac titles on PS5, there’s so much RT technology in place that it allows for various configurations. Meanwhile, many other developers might be in a position where they can now deliver a decent 4K image at 60fps, so perhaps that is the way to proceed with a single mode.

Mark Cerny: Absolutely. I expected diverse opinions. I was upfront about this in the initial video I made, showcasing six games where the first three featured the “ultimate” mode and the other three had entirely different setups.

Digital Foundry: We’ve observed from developers that there has been a wide array of upscaling factors being incorporated into the PSSR process. For instance, in a title like Alan Wake 2, it transitions from 864p to 4K, while in the Insomniac games, it typically scales from 1440p or 1584p to 4K. Does the Sony team offer guidance concerning these upscaling factors, or do you encourage developers to create their own and see what level of image quality they achieve?

Mark Cerny: I’m going to share a bit about the technology we’re developing, and then Mike can discuss how they’re utilizing it. There’s a substantial amount of learning happening there. We support one mode that encompasses a considerable range from 2.5:1 to 1:1. Whether this will remain a long-term approach or not, we’re definitely contemplating it. We’re examining both the low end of the spectrum and the metrics we identify there, compared to the high end. So, I wouldn’t interpret that current 2.5:1 as any indication of where we’ll find ourselves years from now.

Mike Fitzgerald: Yes, I would say a lot of it depends on what you are transitioning from. The TAA we had been leveraging as a studio—and still do—was actually quite effective at around 1440p to 4K, but it struggled considerably as it dropped closer to the 1080p to 1280p range. PSSR performs very well at that level, so for us, it’s a massive improvement and provides us with a chance to approach our work in a completely different manner.

Mark Cerny: Yes, that definitely enhances performance RT.

Mike Fitzgerald: Exactly [laughter]. It genuinely posed challenges for me while playing at home in the performance modes available on the standard PS5. So it’s gratifying to see that quality return…

Digital Foundry: PSSR marks your initial implementation of a machine learning-based upscaling model, and we are observing significant variations regarding upscaling factors. Over the long…
“““html

Crawl, are you undoubtedly looking to enhance these upscaling features or achieve better quality from an original PSSR model? We have heard that there have been quality improvements in PSSR pre-launch, so do you believe this can carry on in the foreseeable future?

Mark Cerny: We are certainly continuing with the enhancement of these ML-based libraries for game graphics. It is true that there are many goals to aim for. Like in the discussion today, I was also reviewing character generation, character extrapolation, ray tracing, denoising, and the like. So what do we focus on and what is our main concern? These are indeed important questions.

Moreover, complicating this is that we announced Project Amethyst with AMD today. Therefore, it is not just us developing the networks; we have a partner collaborating with us. And of course, what the partner perceives about that technology direction and where the focus should be is also essential.

Digital Foundry: K. Along those lines, on PC at the moment, we have various technologies – character generation, AI upscaling, character extrapolation, ray reconstruction, which are making significant advancements to the composition of a game image in real time in a remarkably efficient manner. Are these areas you are investigating for future PSSR upgrades?

Mark Cerny: Certainly. Well, it wouldn’t be PSSR – PSSR stands for PlayStation Spectral High Definition. Spectral is our branding for the ML libraries for graphics, just as Tempest is our branding for our 3D audio. So it will be Spectral something for all of that. And yes, we are definitely looking into what that Spectral something could be, apart from High Definition. However, High Definition is a really great place to start.

Digital Foundry: Regarding PSSR and how it is implemented on a game-by-game basis, developers will likely need to integrate and patch in original versions of PSSR, right? Because you wouldn’t necessarily want new versions of PSSR mixed in without a title update, or is that something you’re considering?

Mark Cerny: Well, there are many intriguing possibilities there. And honestly, I do not know at the moment. It is very nice to think that the game might announce it is okay to update or a user might decide to do so. Just how all of these elements fit together, I’m not sure.

Mike Fitzgerald: From a developer’s perspective, I believe we can view our games in a completely different light quite often. And I know some developers would say, “I don’t want the system to alter what I have shown looks good to me and players.” Other developers would reply, “Oh, your system will enhance my game’s appearance for everyone without me having to do anything? Fantastic.” So there are varying approaches there.

Digital Foundry: With PSSR, you can achieve a remarkable enhancement in image quality while maintaining the internal resolution. Is it a challenge to communicate that this is a significantly improved image even though the internal resolution remains unchanged? I know we are part of this challenge clearly…

Mark Cerny: You are not part of the challenge. I’ve heard your discussions. I’ve spent too much time on DF. However, I’ve heard you mention that we really shouldn’t focus on internal rendering resolution and draw too many conclusions based on that. We should really be focusing on image quality. And I wanted to convey precisely the same message today, that we need to change how we talk about resolution. Perhaps we shouldn’t discuss it at all. The dilemma is when you lack a metric, it becomes very difficult to have that conversation.

Digital Foundry: Yes, we tried to avoid it with a model that could estimate image quality, but it is indeed a significant challenge in terms of communication. Resolution still plays a major role in that discussion, at least for now. One observation we made with PSSR in Ratchet and Clank specifically was that it seemed to improve the clarity and depth of ray-traced reflections. Is that something you might have observed across other titles you are working on, and do you expect that to extend to other games utilizing ray-traced reflections and PSSR?

Mike Fitzgerald: Ray-traced reflections are a fascinating challenge because they are a composite of multiple images in the frame, right? Therefore, with the precision you input into an algorithm like PSSR, you’re limited in how much you can analyze each pixel.

For instance, when you are passing the depth buffer to PSSR to show how far things are from the camera, it cannot indicate how far the window is and how far the reflective object is and how far the object behind the window is, correct? As you turn the camera, you may have some depth information regarding how the window is changing, but perhaps not information about how the reflective surface in the mirror is altering. This is indeed challenging to encapsulate in an analytical algorithm like TAA, thus this is one area we were hoping to see improvements in PSSR as it might have greater potential to recognize this type of issue or adapt to it.

Digital Foundry: With PS4 Pro, the emphasis was on 4K displays. With PS5 Pro, you are achieving a tremendous leap in performance level. Did you have a different kind of gaming experience in mind to deliver to players when designing this console? Or was it more about engineering significant performance improvements and letting developers show players what that looks like?

Mark Cerny: It is indeed a mid-generation release. So the focus is clearly on superior graphics, and it felt like…

“`

The significant three were the consistent topics to address afterward: enhanced GPU performance, improved ray tracing capabilities, and machine learning advancements. Not too complex of a specification!

In my understanding, it has been genuinely extensive to delve into the machine learning aspects of this and gather some insights along the way. Because we all recognize that this is the long-term journey we are embarking on. Thus, it’s a remarkable opportunity to collect all that information with the Reliable, rather than anticipating the launch of a new generation and having to get everything right without any prior knowledge.

Digital Foundry: One intriguing point regarding the PS5 Reliable is that while the GPU, memory, and storage have all been enhanced, the CPU has remained unchanged except for a clock speed increase that developers can utilize. Why wasn’t the CPU upgraded?

Mark Cerny: It’s truly just a matter of where you allocate your resources. I wasn’t observing – and I didn’t believe I would notice – that many titles that couldn’t run at 60. That’s sort of my benchmark. I’m not inquiring, “Do all the games run at 120?” at this point in time. I’m asking, “Do all the games run at 60?” It seems you could count the AAA exceptions on one hand, I think, for games that ultimately couldn’t run at 60. So it appeared that if we focused all our resources on graphics, that would be the right approach.

Mike Fitzgerald: Exactly. I would say as a developer, the day Mark arrives and says, “Hey, we want to present to you what we’re considering for the next hardware,” is an exciting meeting for everyone, and we were more than satisfied with the direction the focus was taking here.

Digital Foundry: Concerning the machine learning hardware within the console, do you see that as a solution for the reason that you no longer get ultimate value scaling on new silicon nodes to any significant extent, such as what historically enabled mid-generation console refreshes?

Mark Cerny: Yes. The data that we had through PS4 Reliable or the like… I wouldn’t say it has reached a ceiling, but generally, it’s about expanding the GPU or speeding up memory. So, as we look toward the future, the advancements will center on ray tracing.