Coming into GTC – Announcements on Data Infrastructure and How the AI Markets are Shaping Up – Infrastructure Matters EP75

This is Textron tv. All right. You are watching and listening to episode 75 of infrastructure Matters because it does the, you know, I'm the only beard on the stage today.

Kimberly. What, what's going on with that, John? I'm not sure.

I mean, I text him, called him slacked him. We're concerned about it. I'm hoping he's not got a problem with We, we are.

I know. com, so I'm sure he's fine. And we, it is Jesse scheduling snafu, and he'll be back next week to help us unload on all of the news coming out of GTC next week.

I'm excited. So, what does the CIO chat, let's let everybody know what he's doing. com.

And now he's even, you know, he, uh, does it on Blue Sky as well, where he basically poses these ccio level questions to audiences, hashtags, ccio chats. If you wanna follow the highest tag, follow the historic conversation, search for that, uh, hashtag. And so then he hosts a day, a a day that they're chatting back and forth.

So it's A lot. Yep. It's a day.

And it's usually, I wanna say it's usually Thursdays, uh, every Thursdays every week is a back and forth chat with the, with the largest CIO community. It's a really great way to look into that community and see what's how to that, what the hot topics are and what their discussion. Well, I'm actually gonna have to follow 'em.

I didn't even know. And the other thing I wanna ask about, because we had a internal meeting talking about some, some six five media kind of things, which is, you know, part of what we're doing here. Mm-hmm.

I'm gonna be doing some new podcasts as well, aren't you? Didn't I hear About Yeah. At least one new podcast is, uh, it's going to be six five on AI hosted with me and, uh, Dave Nicholson.

Uh, we're going to have some of the world's most sm I, I guess I, I, I'm, I'm, uh, telling on myself, because I'm saying some of the world, most smartest people, I don't think that's the right, the, the correct way to say that. But we'll have some of the world's smartest people in AI analysts, uh, vendors, et cetera, interviewing them on the topics of generative ai, agentic ai, and all things ai. I'm looking, I'm looking forward to the new podcast, And I would love to connect you with a group that I was invited into called The American Society for ai.

And it's a beautifully cross organ, cross, um, disciplined organization. Everything from, like, we've got, there's a Microsoft lawyer on there, there is a Lieutenant General, retired Lieutenant General on there. There's, um, senators House of Representative people that are part of it, and little old me, somehow or another, I got invited to this thing because it's, it's by invitation only, and it's kind of like, yeah, I, I don't know if I, I don't know if little old you is that that's a, that's a little humble.

Uh, Kimberly, the, uh, the all, with all due respect, I think if you ever decide to hang up your boots, it's going to be a sad day in the industry, uh, for when and if that ever happens. So, yeah. Yeah.

And we just submit a, um, for an RFI that the government had out for, you know, kind of ideas, certain questions that they had put out, um, as the organization, so consolidated this organization with in terms of knowledge base. So I'm gonna bring those people to you because there's some really smart folks that are on that, that group. I love actually, smart folks, smart questions.

I had Brian Lau, who's a, uh, principal engineer over at a WSI had him on my CTO Advisor podcast. And we talked about the art of the art and science of using AI to do code development. Mm-hmm.

And it was a really great preview to the conversations that we'll have over on the six five channel. So we're, we're looking forward to that. So let's dive into the news this week.

Again, light Newsweek compared, because we're going to GTC and everyone is holding their cards. I think Dale has a pre-briefing at like 10 o'clock at night on Monday, the week of GTC. Everyone is really holding their cards closely.

Uh, we, we have had some pre briefings, but we'll talk about, uh, what we can publicly. Uh, your first story is data infrastructure analysis. IIBM scale.

Yes. So IBM storage scale, not to be confused with Power Scale from Dell, or be confused with Spectrum Scale, which is what it used to be called. This is something that drives me crazy.

The vendors, do you get a new marketing person in and they, they rename everything. It's like, don't do that to me. Okay.

So what this is, is GPFS, sorry guys. This is, this is their, You we're gonna, it'll, it'll, it, it'll, power scale will always be the vnx to me. It just, it just will just just it, it will, but we, we digress.

We're, we're, you know, we're showing our storage by bias, and the greatness in my beard is starting to come apparent of why wives there. So, but spill, Well, lemme me talk grayness in my hair. There we go.

Okay. So power. So I, IBM m Storage Scale, um, is the classic G-P-F-S-G-P-F-S has been the dominant parallel file system used by all the HPC mark, not all in the middle.

A lot of the HBC people over the years, they have simplified the system. So it's a whole lot easier than it used to be. It used to take a lot of probably a hundred students to, you know, put it up.

And now, according to my buddies that used to, uh, run this thing, it's a whole lot easier to implement and install. So now what they're going into, because this is their primary system for ai, um, as you can imagine, this thing scales well beyond most systems. This is like a competes with luster in those kind of areas.

Um, what they're doing is they're adding in connectivity to vector databases. Like we're seeing many of the storage companies doing. So this con a couple sessions ago, we had a little bit of debate to say, is AI a layer on top of the current transaction area or systems?

Is it a separate AI kind of system? The other debate is, do you have vector databases integrated into your, your, um, data storage system? Because you can smooth out your, you know, your AI pipeline, or is it separate?

So they're, what they've said they're bringing, you know, this is bringing out the Vector database in there. Um, for that they have the ability to change it out. So just because it's one Vector database today, if there's another one that's more popular that comes out in the future, they'll be able to change that over to a, a new system.

So it's, they're, they're jumping on this bandwagon, if you will, of where we see this piece going. So now we have, in terms of this space, you've got NetApp that's gonna be rolling theirs out. They pre-announce their Vector database capabilities.

You have Vast that's already there, that's got theirs, theirs out there. So we're seeing, and you know, Dell is doing an integration with a, another company that's doing investor database Starburst, right? Um, they're, they're integrating with other companies.

They haven't announced their own Vector database. Um, so we're, we're saying this, and then we've got some probably announcements coming from HPEI am sure. And because Discover is gonna be coming up in a few months, so I'm sure that they're gonna be jumping down this space as well.

So this is kind of like the pattern that's going on. Yeah. So help me, you know, kind of size or understand the market from a vast data.

You know, we're on a call, we renting, what was that last week? And he said, you know, if, if you want scale, you know, there's no other options other than vast. Obviously the, the HPC folks have been doing scale for a really long time.

Is this a kind of pushback against the vast story and saying, Hey, if you want vector databases, you don't have to go proprietary. You can go with a somewhat open system. Yeah.

So this is definitely a pushback to Vast. I mean, VAST has always claimed absolutes in terms of all their announcements in ter, you know, we are the most, we are the best, we are the whatever. You know, we've, we hear that often with announcements from companies, um, claiming the top topnotch space.

Um, and, and to a certain extent, because they're not a pure parallel file system. They're A-P-N-F-S implementation that is part of the Linux kernel and that kind of thing. Um, they can probably claim a the best there.

Um, when you get to a pure parallel file system like you have with, um, GPFS Luster, um, Panasas is probably another one. Wca, you know, maybe there's, there's some differentiation there in terms of how they scale. Um, one of the things that we're ta they're talking about as well as others are saying that we're going to exabytes, um, in terms of support in terms of these systems, and we have to support Exabytes.

We have to support tens of thousands of GPUs we in. It's getting gonna be really big. I read an article, I can, I look over and grab, grab this piece on it, but was talking about where the financial institution companies are going and how big that data is going to be here in the next, you know, three, four years of what they're expecting it to be.

That all said is, can we afford it? I don't know. So keeping on the vein of storage companies or storage announcements, pure Flash Blade, EXA Pure, pure is still at it with their all Flash systems.

What, what's the news there? Well, just as IS uh, the news with Flash Blade, EXA, um, just as I said, with Vast being A-P-N-F-S implementation, um, that's what Flash Blade has done. So they are PNFS parallel NFS file system implementation.

Plus what they've done is disaggregated the data, note the metadata nodes from the data notes. So, um, this is gonna super speed up the, um, their capabilities. Um, they hyped a bit of their experience with the hyperscalers as we've, I think we talked about last week.

Um, they are doing, they've been doing a lot of, lot of work with Meta. Um, and most recently they announced that Meta is their client that is looking at them to OEM, their direct flash modules. But in this system, they're going with off the shelf technology for the data nodes.

You have to buy their technology for the metadata nodes because of architecture, I'm sure, um, which appeals to your HPC and ai, you know, markets that are there. So, and then eventually they'll roll it out with their own capabilities, their direct flash modules, which are right now coming out at 150 terabytes, or I think they're shipping 150 terabytes. I could be wrong with that.

Um, but yeah, so they're right in the mix of this rollout with everybody else. We'll see, here's some other, other ones probably this next week. Um, but those are two of the big announcements that are coming through.

Yeah, earlier on in the fight of architectures, there was some debate on whether or not, uh, when compute was too expensive, if mixing or separating the data nodes from the metadata nodes had enough value. I think now it's unquestionable with the advancements in, in indexing, et cetera. And the advantages of being able to dis distribute your metadata across large geographic, uh, regions or we, we, within a data center with tens of thousands of GPUs being able to add, identify where the, what the data is, where that data's at, and how to most quickly get to it, uh, within your data infrastructure.

Milliseconds matter. Yes, ma'am. Uh, KIOS is, uh, 122 terabyte drives the, you we just managed mentioned 150 terabyte module 122 terabyte drives.

We've heard these announcements from folks like soy in the past. This we're, we're still at the drive size race. Yeah.

So they're, we we're just gonna continue on doing this. This is QLC. So they're following, you know, soy, soy has got the enterprise class, um, drive that they brought out for the a hundred, 122 kochia is bringing out, the reason why I bring this up is because Oxia, kochia or oxia, I, anyway, another one of those names, I think They're just happy that we're talking about them.

Okay, There We go. Another one of those names, um, in January they announced, I'm gonna look at my notes here. They announced a AI SAQ software.

I'm not sure how they're pronouncing it. AI sac, um, software that they open sourced. And basically this is going to, this is on the drive.

And what they're saying we can do is you can place the index. This software will speed up the access to the index data that you might have on dram, but now you can put it on the drive. It's gonna be as fast or close to as fast on the SSDs, which is, can have some considerable savings.

Now I wanna see the data on this and the performance information. Um, what they're looking at is saying, okay, so that, that index data that's on the DRAM there, that's there, that's really for the rag. Um, you technol rag technology where I'm, you know, using the RAG technology to update the data, et cetera.

So if I can do that and separate that between those two devices to DRAM versus, um, the ssd, it's a whole lot cheaper. Um, yeah, the, we, we've seen this in the consumer space, we've seen, you know, really big SSDs, but with limited dram, if they have, uh, limited dr, then the performance just suffers. It's horrible.

At a matter of fact, you get a lot of storage, and this has been the knock on QLC, right? A lot of, uh, size, but not a lot of performance. So, you know, you get other technologies like TLC, et cetera, but it seems like companies like Sime has solved it.

But that's with, you know, using just enough DR to, uh, do caching and, and improve overall indexing performance. So, you know, I'm, I'm a little curious about this announcement. Uh, storing indexing on QLC absolutely brings down the cost of the drive, but performance wise, I have to imagine there has to be a hit somewhere.

Yep. And so I'm gonna dig into it. I mean, I just kind of looked at saying, okay, so this is really curious.

They're catching up with whatever. So the battlegrounds are gonna be there, especially as we go into exabyte space, but I wanna see this performance and capability. So maybe they'll call our signal six five guys and say, Hey, Yeah, can you, uh, can you help validate these, uh, pretty aggressive claims?

I love the competition in the space. This shows how you can take a single spec. Like we, we know that, uh, you know, we're at 122, we see the roadmap to 2 56.

We see the roadmap. We know that the ability from a wafer perspective to get to 2 56 and beyond is there. But where do these companies di uh, differentiate and innovate and find out, uh, find different ways to eek out just enough performance, just enough price, et cetera.

Where, where do they compete? And, uh, I'm, I'm happy that co co coo Ki Uh, that they're, that they're innovating. I, I think, uh, you know, other than Micron, I'm not sure why all of these, uh, drive companies have such interesting names, Soine, uh, Oxia that they're not, they're not, they just don't roll off the Western tongue, that's for sure.

All right. Moving on to CI, is it CISA or cisa Red Team? Cisa, cisa, Cisa Red team.

So CISA is the cybersecurity agency in, um, the government and it's Homeland Security. And there was a big noise about them letting go of a contract that dropped a hundred people that were considered part of the red team. Um, good guys that are doing penetration analysis.

And so there's a lot of noise that I picked up on multiple trends that can say, this is what, you know, the government is doing. They're firing all these people. We don't have it.

I've been involved with cisa now, I dunno, six years or whatever. There's a local ci, there's a local CISA here. They host, in fact, I'll be down with the, um, them in Denver in a couple weeks, um, where they host a, a multi-dimensional organiza group that comes in and shares information, everything from technology, educators, police, local police, national, you, not national FBI, et cetera, and share information.

But the CSA guys are the, really, the cybersecurity, when you get hit, especially like a bank, they're gonna be involved probably in, in some way. So, so they're connected to all this homeland security kind of work, et cetera. So when they said they fired the red team, it's kind of like, whoa, whoa, whoa, whoa, whoa, whoa, whoa.

What? Um, and the answer is no. And I wanna kind of get out there that that's not what happened.

What they did is they canceled one of the contracts that is a contractor that is doing some work for the ci a red team. The red team still stays in existence. They're still, you know, they're big organization in terms of how they are working through all the penetration.

Um, if you wanna know more about it, you just Google csa, um, government, and you can see all kinds of stuff. So they have lots of education, um, that goes both for, you know, the, the police or the, you know, the, and, and private sector in terms of cybersecurity, et cetera. They do focus a lot on, on consumer, um, initiatives.

Um, a lot of the information there is a lot of consumer, because that's kind of, you know, when you think about the local police, that's who they're really working with as opposed to the FBI's and, and those kind of guys, or, or the on NSA are working on the international problems that come across. So really, um, interesting group, but I just kind of wanna bring that out if you don't know they exist. All right.

So we haven't talked a ton about AI other than the vector databases, but the market is complex, and you have the show notes to talk about kind of this, this complex relationship against a a across all of the different AI markets. You know, uh, the, I I want to, you know, highlight or zero in on the term that I've heard over and over again, the AI factory. Mm-hmm.

Talk to us about, you know, kind of the overall AI market. So I know you've got opinions on this. Last, last, um, summer, um, Dell coined a term called AI Factory.

And, um, Jeff Clark got up and talked about how every company is gonna be building this AI factory. And, and that's why they're hammering down on what this looks like. And that term has now been picked up and it's now ubiquitous.

I mean, we're gonna hear a lot about it at GTC this next week. And, and so then, then we're talking about, okay, so what is this AI factory or what can it be? And um, in some of the presentations that we've had from some these companies, the data companies as well as the, you know, the server companies, they're talking about this market.

And what we're seeing now is there's kind of like these three major markets. There is the super hyperscalers, like an open AI kind of guy. Google's doing their thing, et cetera, that are building the original LLMs.

And then if you go down from there, the next layer down is probably this, this company that is doing, or firm organization that's doing like an AI factory. They're specializing or they're investing heavily in developing either this sub foundation model, let's say a DNA model. Let's say it's an energy model, let's, like, it's an architecture model or a weather model.

They're building these huge environments that are gonna be the guys that are absorbing in the exabytes on the big, huge exabytes. And there may be specialized clouds. They may be a research center like, you know, our, our local CU here that I have in Boulder, um, doing some sort of specialization.

So that's, those are, and that's where the big numbers is. And so for next week, A GTC, this is the market they're probably focused on. Mm-hmm.

This very, very big market that they're saying, this is where the billions and billions or trillions of dollars is going to be exploited, um, because they're building these huge environments out. And then the next layer down is the enterprise ai. And they're the ones that are gonna be doing the rags.

They're gonna be ones that are doing the, you know, the streaming transactional data, respond, give you the customer service capabilities you want the retail capabilities, the development capabilities, et cetera, that we talk about is the implementation of what we're seeing and the results of this. So those are the three markets. And when we start looking at these markets, you gotta look in saying, okay, so what technology from infrastructure matters?

What technology do you need for a infrastructure, you know, an enterprise organization in the local state and local kind of co companies? What technology do you need if you are a, this big AI factory that you're building? And then what technology does the do the hyperscalers hyperscaler spelt?

Um, so that's kind of where I'm gonna start watching this market going to, um, and saying, okay, so who are you talking to and how are you talking to, and what, what are the demands of these people? Yeah. So this reminds me, I'm, uh, Monday next week at the next week of this recording, I'm headed to GTC, obviously, but there's kind of a pre GTC unconference hosted by a MD.

It looks like it's hosted by a MD. It's, the idea is beyond cuda, this idea that, that you must challenge the notion that NTO end Nvidia is the solution for all of ai. And it's very difficult at this point to say any one solution is the, is the answer to all of AI for any of these markets you just mentioned.

So for example, at the most basic layer of compute, we've talked about some of the, uh, friends or customers of Dell, the nature, fresh farms of the world that's doing some amazing stuff around growing tomato plants inside of, uh, greenhouses in which they collect data off each and every plant, analyze that data and try and prove their yields by one or 2% each, uh, individual crop. All of that has been done on cpu, just mm-hmm. Good old fashioned, uh, VxRail nodes with no GPUs in there, and not really using much Nvidia technology.

Whereas when you're talking about, you know, that, that AI factory story, that the, especially with Dell, that's very an, uh, that's very much an Nvidia heavy conversation. And the, the one piece of news that I did mix mi miss this week that it's relevant to your highlighted is meta announced that they are testing and they're phasing in their own, uh, gpu, I'm sorry, XP, which are, you know, accelerators, but not full GPUs. And they are developing their own custom layer of software to, uh, be able to ingest all of this data and do all of the training.

So it is an amazing, uh, I think, technical landscape across all three, uh, of the markets that you've highlighted. And I, I think, you know, as I talked about earlier last, this last year, we're gonna see within every industry there's gonna be this sub or multiple sub foundation models is, you know, it's kind of like going back when I talked to the CIO at Swift, what they were building, the SWIFT is the transaction people, you know, this is well over a year ago they had built the LLM that's looking at wire fraud, and it's taking an ingesting all of that information. And if they can just drop the wire fraud, you know, a couple percentages, you're talking trillions of dollars here.

So it's just a fascinating market, um, of what is getting built out. And then you, my thinking process about, okay, so if SWIFT can build that for the international transactions and everything goes through them, then if you share that, can that data then be shared with the other banks to implement on top of their, you know, web services capability, um, that trains their data, et cetera. But they don't have to build the big huge model.

So that's was one of the things I was looking at, is saying, okay, so this is where we go with this, you Know? Yeah. What happens when they can create, uh, uh, signatures almost that allow and be able to push those down to other banks' AI models so they can now, uh, uh, adjust the sensitivity based on their risk profile, their client base, et cetera.

But what happens when you get the strength of a swift to be able to do what we talked about at the onset of this, we're talking about GPFS and, and vectorizing, uh, the data and absorbing all of these exabytes of data. What happens when you have a provider in the middle of that that can take that heavy lift and then go down to that third rag layer that, as you described it, and able to implement off the shelf technologies that help them to actually implement, you know, some of these techniques. Fascinating time.

Uh, and it's, it's absolutely why infrastructure matters. But before they get to that, my one piece of news is Kubernetes solving the problem is finops solving the problem that we are hoping to solve in their enterprises over at CIO dive. They're challenging this notion that over provisioning is done.

If, you know, if you don't know what a container is, you're probably listening to the wrong podcast. So there's that. But there's the idea that, uh, if you have this high level orchestration layer, you can better not only control the scaling of your applications, but the resulting cost.

But developers are developers and do developers care about cost? The, it is a invisible thing to them. This whole movement around DevOps and this idea that developers are the single source of both, uh, solving the business challenge of writing the code and controlling the infrastructure was always a bit of a pipe dream.

And when you adopt technologies like Kubernetes, this is not, you know, this is not a fix all, it, it, it's not gonna solve every problem in your enterprise. You still need domain experts to really help tune in, negotiate between physical groups of people analog. The analog is still the challenge.

You still need to help educate developers, control costs, put systems into place that, uh, help to, uh, meet the, the end goal of skelly applications at a reasonable price. It doesn't matter if these applications are on, on premises or in the public cloud, the issue is still the same. So what you were saying is that they're over provisioning server compute, storage, probably in storage.

It, they always have. Um, and they're over provisioning in the cloud, despite them having a lot of tools that would enable them to keep it skinny. Yeah.

They're, they're just looking to solve the core business logic challenge, right? Uh, how do I, how do I process this many orders and this amount of time? And the, the, the, the cheat and enterprise it, and it in general has always been throw more compute at a problem.

It is much easier to throw more compute at a problem than it is to optimize code. Again, when we're talking about AI generated code, your prompt is probably not including optimize for the amount of storage ram and compute that I use to solve this problem. Your lens is typically help me write code that solves this problem.

It doesn't have to be efficient. Uh, and help me, you know, write a yamo file that, you know, scales to the enough compute to accommodate this code. So are you thinking that we're gonna use AI to help do that a little bit better?

I think we are going to use ai. We're starting to see AI creep into, uh, conversations around network design, network configuration, finops obviously the folks over on the security side and so have been way ahead of almost everybody using LLMs to, uh, help, uh, uh, triage incidents and, uh, really get ahead of this thing. I think AI is the way, right?

I can just take all of my logs in theory. I can take my log files from my Kubernetes clusters and my costing data from AWS Azure and GCP, upload that to a multi, uh, expert model and say, Hey, optimize for cost. Eh, I might be oversimplifying it, but that's the overall idea.

Yeah. Okay. Cool.

So before we close out, I have a really important question. Okay. What's the logo on your shirt?

Oh, Show I'm doing It is, it is, it is the DePaul University. DePaul University Blue Demons. It is where a lot of my personal money and my son's money, and then my money again for my son to go to school.

Uh, last night, uh, of this recording and the Big East tournament, they gave credent a run for their money. Uh, they end up, uh, losing in double overtime by four points. It's Just the round ball.

Heartbreaking. That's the, that's the round, that's the ball. That's the game where the ball, the ball is round.

You throw it into a basket. That is, that is the game. It's the only game that that's at DePaul.

We don't have any other game, uh, uh, sports ball games that anybody else pays attention to. It is no hockey. There is not that I know of the, the, there's, I'm sure there's a tennis team, I'm sure there's a golf team.

Um, but we have no football team. So basketball team is the one sport in Chicago that we cannot get good at again, which is a crime because we're in Chicago. But, you know, I digress.

You, you took me there. You took just, I took you there. I've been, I've been thinking it for the last half an hour.

What's that logo? What is that man, that is, that is the great University of DePaul. You know what?

The computer science program, it's much better than the basketball program. I'll just put it, I'll just leave it. I'll just leave it at that.

All right. All right. That's it for this episode of Infrastructure Matters.

Make sure to stay tuned next week this, we'll have the output from GTC. It'll be more news than we can digest in a half an hour, but it'll be worthwhile in preparation. Again, I suggest you put this onto a thumb drive, throw it onto the street, and see how many people pick it up and put it into their computer, and listen to it.

Talk to you next episode of Infrastructure Matters. Okay.