NVIDIA Vera CPUs Target the Next AI Infrastructure Bottleneck

Hey guys, thanks for the throw. We're here with Hannah Coutand, who's the director of product marketing for the Vera CPUs from Nvidia. And I know we all think of Nvidia and GPUs, but they also have a line of CPUs, and well, these are different than your average CPUs.

And I'm going to let Hannah explain what type of workloads we're going to see on these things and where they fit in the universe of processors. Hannah, welcome to the show. Thank you so much, Mike.

I'm excited to be here and excited to share the story of Vera, yeah. So start at the beginning, because I think most folks don't really know a whole lot about Vera CPUs because, well, they're all busily watching what's going on with these GPUs. But I would argue that CPUs are going to be carrying more and more of the inference workloads.

Is that fair? That's fair. As well as agentic sandboxing, reinforcement learning, data processing, simulation.

Those are some of the other workloads that we think Vera is pretty great for as well. Right. There are a lot of CPUs in the world.

What makes the Vera CPU different? Yeah. So you did say you wanted to hear the story of how it's different.

I'm happy to tell that story a little bit, but it takes a little bit of a retrospective. So if you would indulge me, I'd say that in order to understand why Vera is the way it is, what makes it special, you have to go back to the pre-cloud era. So back in the pre-cloud era, right before the hyperscaler booms, let's say 2014.

At this period of time, data center CPUs looked a lot like client CPUs, which is that they had high single-threaded performance, core performance, but low throughput. And so, if we put ourselves in that time space, Intel had 90-plus percent market share. Their Intel Xeon E5 V3 Haswell, launched in 2014, had 14 cores, and 18 cores max or something like this.

And CPU vendors at the time touted faster cores every generation. There was this focus on per-core speed. And so that was the data center CPU of the time.

And actually, notably, no one customer type really could dictate CPU design. But then when the hyperscalers came into the picture, and the period of time when the hyperscalers became the hyperscalers, they really changed CPU design dramatically through their scale economics. They grew through scale.

They understood that by offering their customers the ability to scale their application across a global compute footprint, you deployed Kubernetes, and bam, your application gets scaled across your compute footprint. They understood that if they could offer that value prop to their customers, they could scale their business. And so they used scale economics as a way to grow, and then they grew, and they asked of CPU vendors at the time.

There were sort of three things that happened in this period of time that really influenced CPU design, and I will get to why Vera is the way it is, but I feel like this is kind of an important retrospective to understand that. So number one, the hyperscalers asked CPU vendors at the time, "Please give us as many high core counts, high throughput as possible," because CPUs and core counts were the essential building blocks of this sort of scale dynamic. Even if you look at a CSP's revenue growth over this period of time, let's say 2014 to 2025, that revenue growth is growing 16 X.

But their instance pricing over that same period of time is actually falling. So the actual instance pricing over that same period of time declined by about half. So if you take the basic equation of business, which is that revenue equals volume times ASP, if ASP is going down, well, volume must be going up.

And so we see that manifest in core count growth over this time period. So core counts actually grew from Haswell at roughly 14 cores in 2014 to Intel's Emerald Rapids in 2024 at 64 cores, and now AMD EPYC Turin is 128 cores. That's 9 X core count growth.

So just to make the point that you can actually see how much the hyperscalers influenced CPU vendors and this need for core count growth and throughput at that time. So number one, core count growth and throughput was kind of a big design point. Second design point was around cost, and that CPU vendors, in order to drive more optimal costs, employed, especially among these x86 vendors, chiplet architectures.

And chiplet architectures are composable, and when you compose them together, it leads to higher core densities. And then chiplets, because they're smaller and easier to manufacture, actually lead to better yields and therefore lower costs. So that's kind of a second influence point on CPU vendors and design.

And then the third is that there are kind of cloud hyperscalers, or there's two buckets of workloads, batch workloads, which are very latency insensitive, but throughput intensive, and then front-end workloads, which you think web serving, ad serving, search. Those were, in theory, very latency sensitive, and in theory would value single-threaded core performance. But the way hyperscalers treated this sort of paradox of having these two buckets of workloads with very different requirements, they used scale-out techniques to handle those front-end workloads.

" And so they used scale-out techniques to address that, and therefore converged to a single design point around Core counts, throughput, lower costs, right? So telling that story because when we exit the cloud, this hyperscale growth area, and now we come to the agentic AI moment. Agentic AI is actually asking for a different design point for CPUs than what the cloud era gave us.

And so going back to your question then, so agents are actually asking for, because agents work in a loop between the GPU and the CPU, and they loop sometimes hundreds of times between the GPU and CPU, and they access memory multiple times. And so the GPU is where they reason, and the CPU is where they run a bunch of sequential tasks. So tool calls, API calls, scripts, opening a web browser, like a bunch of sequential tasks.

And if you actually look, and that loop is super data hungry, and memory intensive. And so agents actually want faster cores to speed that loop. And actually, you can't get a faster loop by more cores.

You actually need a faster core in order to speed that loop. You want high memory bandwidth so that you can access that memory, and your agents are not waiting for the data. They're fed consistently with data.

And then thirdly, you want consistent latency so that you can return to the next reasoning step on your GPU as quickly as possible, and you keep those GPUs, highly valuable assets in the AI factory, as highly utilized as possible. So, those three things are what agentic AI needs. And so that is a different design point than the cloud CPU of yore.

And so that is a design point that is rather high single-threaded core performance as opposed to low single-threaded core performance, but high core count, which is what the cloud era gave us. And so that's why Vera actually excels on all three of those dimensions, high per core speed, high memory bandwidth, and exceptionally low latency. And so, sorry, that was a bit of a long-winded answer, but I wanted to make sure I thoroughly answered the point of why is Vera different?

Why do we need this type of CPU? What makes both it different and why is this moment different? One of the issues that people are struggling with is utilization, and it's been an issue for CPUs and GPUs.

So does this architecture for agentic AI workloads or any other type of workloads, is it fundamentally going to be more efficient so that I am consuming less energy, I am getting more throughput per watt, or whatever measure you want to use? But can we think about this differently, or what are the downstream benefits to folks? Yeah.

Utilization efficiency is certainly a hugely important topic of conversation today. I think this is also a dimension on which Vera, as a differentiated CPU, excels. Vera, in order to deliver that exceptional memory bandwidth, we have our own NVIDIA-designed LPDDR5X memory subsystem, and the LP in LPDDR5X stands for low power, and it was actually a memory subsystem designed for mobile use cases.

And so, we designed a module that can be employed in data center class CPUs that then offers that high efficiency, that low power, but then delivers the high memory bandwidth you want in data center class use cases. And so what this means is that on a efficiency basis, it draws an order of magnitude lower power, it draws an order of magnitude lower memory power than competitive memory subsystems. And so I think we recognized the importance of driving high efficiency, higher utilization, and so that was an important design pillar of Vera and the CPU that we wanted.

One of the things that people at least have a perception of is that these are expensive processors, and they're going to be for certain higher-end workloads, but are they going to be usable for more of an affordable general purpose? What does that price-performance curve look like? Yeah.

I think we purpose-built Vera for the agentic AI era, but that doesn't mean it's a one-trick pony, if you will. Vera is also exceptional for data processing, simulation, and we believe that general-purpose CPUs, as they have been for a while, there's tremendous fungibility across data center and enterprise use cases. And so I think we're just at the tip of the iceberg on the types of use cases where Vera can shine.

The ones that I've listed, those are the ones where I think there's broad applicability across the use cases of both AI natives or enterprises who are embracing agentic use cases, but then also the fungibility once you have that type of CPU infrastructure to extend to data analytics and data processing as well. Yeah. And I think one of the other things, and you touched on it, but I'd like to get a little deeper, is that a lot of these AI workloads are spiky.

They're not as predictable as certain other classes of workloads, and it sounds like we need a CPU architecture that's designed for that kind of interaction and is going to allow us to support those workloads without having, for lack of a technical term, all those CPU helpers that you previously discussed that people had to compensate for the architecture decisions they made, and I won't have to lean so much on cache and CDNs and all this other stuff. Yeah, I think we are in the early innings of knowing what the agentic workload looks like. From what we understand now, it's highly nondeterministic, right?

You can't necessarily predict the path an agent is going to take. So I think your characterization of it being spiky is pretty apt. But that, I think, also reinforces the need of taking a CPU that really hits the design points needed for high performance for agentic-type workloads.

And so, I think perhaps we'll get to a point in time where we'll have a better ability to characterize an agent's path, as you will, as it completes its work and its task. But for right now, it's very nondeterministic. And so, it's going to loop many times.

It's going to call on lots of different tools. It's going to go back to the GPU multiple times. And so if you can imagine a decision tree, you get a prompt, and then the branches of the tree just become so broad.

You want to be able to make sure your agent can just process as quickly as possible. So that's why you want that sort of per-core speed, high memory bandwidth, so that your agents can be tapping into memory and data as quickly as possible and keeping the latency as low as possible through, in the case of Vera, through having a monolithic die, where you can have high core-to-core bandwidth speed with very low latency. Those are kind of the characteristics you want.

So I guess in a way, as the workloads, as they are spiky, as they are super nondeterministic, knowing that there are certain performance characteristics you'll need for your agentic workload, making sure you take a CPU that really hits those, I think is going to become increasingly important until such point in time we really have a better understanding of, again, that workload path. How accessible are these processors going to be? Because I think a lot of the frustration people have today is that they're ordering GPUs for two years from now because they're trying to figure out, what am I going to need, and they know they're going to be scarce.

So how many of these Vera CPUs will there be out there? Are there manufacturing partners, and what does the supply look like? Yeah.

We're in the early innings of Vera's adoption, but we're super excited by all of our ecosystem partners who are embracing it across cloud, both hyperscaler, CSP, as well as AI cloud partners, both of whom are embracing Vera. Super computing centers like Los Alamos National Lab, and PAC, and NERSC are also embracing Vera. Then across our broad ecosystem of Dell, Lenovo, Supermicro, HPE, these are all partners who we've celebrated as being part of the Vera journey for us.

But in terms of how many we'll have out there, to your question, still TBD, but we're super bullish on the CPU markets. Vera, given that it was our first custom design, and we made that investment in the custom design knowing the importance of this moment, and hence why we're also super excited about sharing the story of Vera and why it's different, because we are so bullish on the opportunity and what we see. So I'd say it's hard to pin a number on it, to be quite honest.

Last question, though. What is your advice to people who are building these AI applications and agents today as they kind of look at these different processor architectures? Because I think they spend a fair amount of time trying to optimize for different processors.

So, as you kind of look at what they're doing and where Vera fits in that equation, what would you tell them? Yeah, I would say, it sounds perhaps overly basic, but do your homework. We have very proudly been as transparent as possible on Vera.

We published a white paper with all of the SPEC 2026 scores for Vera and all of the how it performs on all these sub-scores. We've been very transparent on how well it performs for these different use cases, where we've actively worked with partners like Perplexity and DeepInfra, and there's a set of more partners to come, where they've benchmarked Vera, where we've given them access to Vera in a lab. They've run Vera for some of their everyday coding workflows, and they've showed how well it performs.

So I guess I would say understand what you need deeply, what workload performance characteristics you're after, and then go explore what's out there. And hopefully, we have empowered you with all the information you need to make the right decision on the infrastructure you need to power your AI factory and your workload. But we endeavor to be as transparent as possible, hence why we, as I mentioned, published that white paper, detailing all the technical specs of Vera as well as how well it performs on the latest SPEC 2026 suite.

All right, folks. Well, you heard it here. There's a new kid on the CPU block named Vera.

Check it out, give it a chance, and let us know what we think or you think or what Nvidia, of course, I'm sure is looking for some feedback from everybody and anybody. So by all means, write some software and see what happens. Hannah, thanks for being on the show.

Sten, well, thank you so much for having me. All right. And back to you guys in the studio.