Overcoming Energy Challenges in AI with ScaleFlux’s JB Baker

ai video series. I'm your host, Mike Beard. Today we're talking with JB Baker, who's vice president of Product for Scale Flu, and we're having a chat about, well, AI and energy and the environment and how might all this stuff actually come together.

JB welcome to the show. Thank you, and great to be here. A lot of interesting promises have been made of late, including a Project Stargate, but every time I turn around lately somebody is pledging billions of dollars to go build data centers for these AI initiatives.

And I cannot help but wonder, um, are those things realistic? 'cause last time I checked, it wasn't so much that we didn't have enough or say power. We just don't have the ability to distribute it where it needs to go.

And so can we actually build these things. JB what's your thought on where are we in this adventure? Yeah, that's, uh, as I've been looking at this part over the last couple of years, um, yeah, uh, getting the power to the data centers is, is a tremendous challenge.

Um, and, you know, we see, uh, I've seen plenty of information about, uh, data center build out plans being delayed because they're still fighting through the regulatory issues to get local power plants put on, brought online or, you know, more power brought to the grid. So that is, that's definitely something that has to be done hand in hand as we are, uh, as an industry planning out these, uh, these massive build outs for data centers to, to support the, the demand for AI and other compute functions. And how much do environmental concerns plan in this conversation?

'cause every time we kick up a new data center, we've kicking off some carbon, but to what degree are the data centers contributing to global climate change, and how might that be more complicated than it already is? Yeah, I mean, data centers are depending, I see different stats, uh, but you know, they somewhere from, from high single to low double digit percentage of total worldwide power consumption. So they're a, you know, a massive contributor to the potential for, for carbon, uh, you know, emissions.

Uh, I know that companies like, you know, some of the major hyperscalers like Microsoft and Amazon and, uh, meta and Google, that they have renewable energy initiatives to try to make their data centers net zero or, or or negative in terms of total carbon. Um, so those initiatives are there. Uh, I, well, it, it's kinda governments are, are playing in that as well.

You know, the EU seems to be a little more aggressive on pushing for, for those, uh, carbon neutral and, and carbon improvement, uh, capabilities. And then other GOs it, we seem to go, seem to push for it and then back off and push forward it back off. So time will tell on that.

It's hard to say where we're gonna be in by the end of the decade, but, um, there's a lot going on here and not to mention the least of which is we are gonna see new classes of processors, right? Not everything needs to be a GPU to run these models. So will we get more efficient in terms of the infrastructure that we're deploying as we go along?

And will that make a difference? Yeah, that's a, that's a great direction. I mean, efficiency is crucial to, uh, to meeting the demand that we have, that this massively growing demand for the compute and the data generation, data storage, data movements.

So, uh, yes, absolutely generation over generation, the individual components are becoming more efficient in terms of how much work they can do per walk. You know, it, it's scale flex. We we're focused in the storage and memory domain.

So like within our components, every generation is, is being able to provide twice the amount of performance, twice the, the throughput of data transfer, uh, per watt of energy consumed. And if you listen to the NVIDIA keynotes, um, I don't remember his exact, uh, jenssen's exact percentage or, or number of, of improvement there in terms of, uh, flops per watt. But every generation of Nvidia, GPU is massively more power efficient, though they're also massively more power debts, right?

You're, each processor is consuming a lot more power, but they're able to do, say multiple times the amount of work per power per wat. Uh, and you're, you're definitely right there on, it's not just gonna be, uh, the general purpose GPUs, right? That's, um, the, the black walls and, and that class of product.

Uh, 'cause just as if you, if you look back several years, we, there was, uh, a migration or a fragmentation of the, the general purpose processing capabilities. You know, you had the X 86 processor as kind of the standard in the industry, right? But then graphics processors came online to handle specific tasks in a much more efficient manner.

The same thing is starting to happen already, even in the the GPU segment, even for ai, where you're gonna have, rather than just general purpose GPUs, you will have, uh, LLM specific GPUs. You will have inference specific GPUs and other types of tasks. You're gonna have processors that are optimized for that, for those tasks.

Uh, a couple of examples already. You see meta investing in their own, uh, internally developed processors that are focused on what they care about most, the LLMs, uh, rock as well, uh, offering GPUs that are more focused. And their claim is that they're able to handle the LLM like a chat GPT, um, request much more rapidly than, and much more efficiently than general purpose GPUs.

So you'll see that fragmentation to enhance the efficiency of the processors. And then other components in the, in the industry are gonna have to step up as well. I mentioned the storage components becoming more efficient.

Um, there are innovations in the memory domain to allow for more capacity and more bandwidth of memory attached to each processor, um, or each core within those processors to ensure that those processors are fed with the data that they need to be productive with all of the energy that they're consuming On the software side of this equation. Not everyone in these AI models is the same, and some of them are gonna be smaller than others, and others are gonna probably be even larger than the ones we have today. So, yeah.

Um, can we, how does that spectrum kind of play out in your mind when we see more smaller ones in the future? Or, you know, people are talking about, uh, making AI smarter, which might involve some really huge language models. So where are we?

Yeah, I, I think you're gonna s you're gonna see both ends of the spectrum, right? You, you will see that the massive LLMs will continue to, to grow larger. Uh, there's plenty of, of graphs out there of, of the doubling of inputs or the, the exponential increase in the amount of data that gets put into training that next model to make it more generally intelligent and more generally applicable.

Uh, but at the same time, you, you're gonna see things where, hey, I don't need to have, uh, my model be able to answer every question and act like every different type of profession. Maybe I need my model to act as a, a patent lawyer. Uh, and so that dramatically constrains the, uh, the scope of the inputs that you need to put in there to get to that same level of accuracy or, or value out of the model.

So yes, I, I believe you will see more and more of these, uh, very focused models that, uh, that people will be able to, to train on much smaller data sets, and then those will be more efficient in their searches and their re their responses as well. Do you think all this AI concerns about energy is also gonna wind up becoming a political issue? Because as the data center folks increase demand for electricity might exacerbate a limited supply and the price of electricity goes up and suddenly, you know, all the local neighbors are up in arms?

Yeah, I'm, I'm going to, I'll try to shy away from anything, any true political statements, but, uh, the absolutely, it is a challenge and, and the, the knot in my backyard or, or nimby, uh, perspective will come into play. Uh, I mentioned earlier you're already gonna, you see that already, particularly I saw a lot of information about in, uh, Northern Virginia arena where there's sort of a data center alley and they're struggling to, to provide enough power there. So, um, in order to asage the concerns about data center power consumption, pre putting pressure on the grid, and raising overall power rates for you and I as consumers in our households, uh, there is definitely gonna have to be a, uh, a hand in hand plan of, Hey, as I, as I pro project putting out this, uh, this data center, I'm gonna bring online additional power.

Uh, you know, you see like the, the three mile island thing be where three mile islands will be brought back up to provide the, the data, the power to, uh, supply. I think it was Microsoft's data centers that they're building in that region. Um, and then even in the Stargate, uh, the proposed data center that in Texas that's gonna be, um, what was it, 360 megawatts that they are co-planning power, uh, generation facilities to, to provide that power.

Uh, 'cause otherwise that's 90,000 homes worth of electricity that you're, you need to supply. And I don't think there's that excess in the grid as it stands. We also see people talking about various forms of green slash clean energy, and some people wanna build, uh, many nuclear reactors that are essentially dedicated for data centers.

I'm not quite clear how those things work or how feasible that is, but how much can these alternative energy sources make a difference, Make difference? Uh, it can make a huge difference. And, you know, there's not one magic pill or one panacea for solving this energy, uh, de demand.

It has to come from multiple sources. Um, you know, I, I have, I have seen many initiatives from like, like the hyperscalers in the US where they're, you know, they're combining geothermal with, um, with solar, with localized, uh, fossil fuel based, um, energy, consum, energy, uh, creation. And as you mentioned, the those small nuclear reactors that are gonna be more sized specifically to provide the, uh, the electricity for those PowerPoint for those data centers.

That's another aspect of the solving the, the, the total, um, supply. And even the going, going back many years, the location choice for where you put the data centers, um, you know, they've, they've chosen colder environments in many cases so that they could leverage just the, the natural environment, uh, to help them with the cooling. So absolutely, it's gonna require many different types of power generation, uh, to, to help minimize the, the carbon footprint that gets put in place.

So what impact will all this have on the cost of ai? If the price of energy is significant, and I have all this infrastructure that's out there, is AI gonna get any cheaper? Or is it maybe actually gonna get more expensive?

That is a, uh, you know, a lottery question, I guess. But, um, you know, in terms of the, the cost of performing an AI task, I, I can't, I can't wager in any direction other than that will decrease, right? The, the cost of, of performing a, a chat GPT task or, uh, doing, uh, uh, as Jensen referred to it as the, the virtual factories of modeling, you know, how should we set up our warehouse or, or, or testing these models, though, that's gonna come down, absolutely, but that does not say that the investment levels and the amount of dollars that are poured into generating AI capability would decrease.

I, I still see that as that's, that's on the rise, because as we, as we decrease the cost of, of performing an individual task, you know, just following economic, you know, supply and demand, we will find demand and ways to, to consume that. And we'll find more and more complex tasks to put out there to, to utilize still those large clusters of GPUs. So what's your best advice to folks as you kinda look at all this stuff and they all sit there and kind of try to figure out their own plans, but, um, should I just like pick a handful of projects to focus on for the short term and, or am I, is it just gonna be, you know, pedals in the metal and here we go?

I think, You know, that's gonna vary quite a bit by, by industry and job function, but, uh, but absolutely you have to be finding ways and looking for ways in which you can leverage AI to improve your organizational efficiency. Um, you know, it, those who don't will fall behind, will, will be falling behind. Um, you know, and even as we look at, uh, you know, from my perspective as a hardware component and, and software, uh, developer company, we have to start leveraging AI in the, uh, the development of our next generation components.

Um, and leveraging AI to help us accelerate the, the generation of code and generation and testing of the code. Um, that's an area that I, I personally, I believe that there's a, there's gonna be a, a tremendous value there in terms of being able to test, um, and find the bugs and find the root causes for cha problems in, in hardware, uh, and chips as well as in software. Um, 'cause having been in this industry for 20 something, uh, years, then, you know, I just re I think back to all of the problems that, that we had as, uh, not just scale flux, but at Intel and LSI and other companies of finding a bug in the hardware and fixing it.

You know, what is that root cause for that? Something that appears so, so, so infrequently it, it's hard to recreate the conditions. And AI with these virtual factories has the potential of being, making it where you can recreate those, uh, those conditions much more rapidly.

Hey folks, the genie's out of the bottle. I'm definitely not going back in, but as is always the case, you gotta be really precise about what you're wishing for or may not turn out exactly how you imagined. Jv, thanks for being on the show.

Great, thanks. Appreciate the time. All right.

Thank you all for watching the latest episode of the Techstrong AI video series. You can find this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next day.