Techstrong Gang – April 29, 2025

Hey, everybody. It's RSA week, and we're talking about, well, what else? AI security.

We'll be back in a minute. Hello everybody. I'm Mike Baard, and welcome to Techron Gang.

We got an awesome lineup, as usual. I'm gonna jump right into those introductions, but it's gonna be a lot of AI and a little compliance, so stay tuned. We got a broad mix of things.

Guy Currier, you were the furthest away Going from outside in. Yes, I am in Dublin for the Open Compute Projects EMEA Summit, which starts today. All right.

Uh, it seems like we constantly have people in Ireland as of late, so I will tell you that I'm jealous, but hey, I Was just there, so I know you, you jealous. You were just there. You shift out.

I shift in Bonnie's Nation. Go Word. Oh, I'd love to go.

I've never been. All right. We're, we're, we're trading places, as they say.

Also joining us today, far away, but in an entirely different direction. Tracy Ragan is hanging out in New Mexico, I assume, correct? I am.

Hello all. Great to be here. Always great to have you on the show.

Then also joining us, nearer to me, at least here in New York, is Steven Boskin, who I think is in Ohio. Is that correct? I am indeed.

Currently in Ohio. Uh, where the, uh, we just got a, uh, red Sox player steal home yesterday in the game, which is always a fun occurrence against the Cleveland Guardians. I saw that.

I think some of us refer to that as embarrassment, but what do we know? And then finally, of course, we have Bonnie Schneider joining us from Florida, as always, to give us the latest and greatest Echo Insights report. Bonnie, how are you?

I'm doing great. Thanks for having me on. All right.

So everybody who's anybody is in the security space, at least is descended upon RSA this week in San Francisco. And one of the hot topics of the day is, well, the insecurity of these AI models, there are multiple ways to trick these models into coughing up sensitive data. And recently, there was a report last week from an outfit called Hidden Layers that basically showed a way to bypass every AI guardrail and safety protocol there is, which, uh, may give us all a little bit cause for pause, but guy, you've been kind of kicking around this sector for a little while now.

Are these things fundamentally secure, or is it just a matter of time before they blow up? Uh, I think they're fundamentally insecure. I, I don't know how you, I mean, I'm not a data scientist.

I have never trained a model, uh, at least not a foundational model. Um, and I had to use other tools to do it. That said, um, the principle behind, um, AI and AI training, um, involves a fair amount of mathematical trickery, and I mean, trickery in the best sense, um, meaning, uh, wouldn't be possible to have the tools that we have today without it.

Um, that's point number one. Point number two, at least, you know, when, when I take in this news is that, um, it's all, it's called artificial intelligence. I've started to put the quotes around the I a quote, I, um, because it's not intelligence, it's a simulation of intelligence.

It's extremely sophisticated, one with all these billions of parameters and all that sort of stuff. So it appears to be, uh, reason not reasoning exactly, but formulating things and cogitating and so forth. But in fact, it's not.

And so any system anywhere forever, um, including the future quantum systems, um, are going to be, uh, you know, attacked and exploited or found exploits found one way or another. There's lots of money to be made there. No reason why that wouldn't be done for a system like ai.

Okay. So how does that take us to today? Um, the thing is that, um, when these are trained and when these have been trained, I don't know if there actually can be security elements, um, uh, uh, as part of, uh, the training.

And so the result is you have this complex, uh, system that's been trained that you can't exactly define what it is and what it's doing, only what its outcomes and outputs are. Um, and so, uh, these attempts to break it, um, security wise, whether by black hats or by white hats, in this case, white hats, um, are inevitably gonna be successful in some way or another. And this prompt injection attack, I don't know if we can really, uh, turn it up to 12, but we should be turning it up to 11 by now, already 12, if possible.

Why? So, um, this particular prompt injection attack, the example that these, the prompt, prompt injection attacks, uh, essentially put a very weird, uh, not human input in the prompt and context into the prompt. But listen, it's not a reasoning model.

It doesn't look at it and say, oh, that doesn't look like a human. Put it in. It just takes whatever it took and outputs whatever it's gonna output.

And in this case, it's outputting sensitive information, like information about nuclear devices, information about, um, biohazards, that sort of thing. The kind of thing that teaches people how to make bombs and stuff like that, that supposedly these systems are supposed to protect against. But because of the weird input put in that a human normally would not put in the model doesn't know any better, and even though supposedly it's supposed to filter its outputs and say, I'm not supposed to tell you that information.

It tells it to you anyway. Why is this so dangerous? I mean, it's one thing, as the example provided, it's one thing to give medical advice when the model is not supposed to give medical advice.

It's quite another thing. If it's an AI you're using in your enterprise or your secret government agency or what have you with stuff that can really endanger as little as your own competitiveness or as great as national security and some prompt injection attack, just reveals that to all comers. Tracy.

Um, are you comfortable using these things? 'cause you know, when I talk to some security people, they'll tell me that these models and agents, they keep a secret about as well as a five-year-old. And if I've got sensitive data even exposed through a rag connection, suddenly my IP is available to everybody.

Well, you know, I used to be comfortable with it. And then, you know, when I first learned about Gelb brakes, I started thinking twice, right? Um, because I mean, there's actually an application online.

I talked about it once, that you can learn how to write good, uh, prompt injections or gelb brakes, uh, to get into these systems. So we're, we're just so new in this, in this space. I wouldn't say I'm afraid to use it, and I'm not going to, you know, we're, we're, I'm a technologist.

I run a open source community, they're gonna run down this, uh, road as fast as everybody else. We just have to start getting smarter about this. Um, ironically, I met a woman over the weekend who works for the government, um, and, uh, securing these models around, uh, nuclear energy.

And she's losing, uh, she won't be there in, uh, two months, or job is over. Her contract's over, they don't wanna pursue it anymore. They don't wanna worry about it.

That's what she was told. It's not a focus for our government. Um, so we're being asked for the, what's happening is the private sector is being asked to start working on these, um, these issues.

And sometimes the reason why we have government is because government will work on things. It's not profitable. So we find ourselves in a situation now where we are going to really be pushing these technologies, and we don't have a government who's going to be worried about how secure they are.

And we have to make sure that the private sector decides that they wanna spend money and, and, and sorting out how to, uh, identify how these jailbreaks or prompt injections are done, the, the what catastrophes they could create. Uh, and we don't have it. So it is concerning, and I'm hoping that the public, uh, or I'm sorry, the private sector steps up and starts spending money on securing, um, uh, these models.

And we haven't even thought about, you know, I, I've thought a lot about the, the implications of, uh, these mc, uh, MCP, uh, ai, uh, APIs, because now what we're doing is we are creating, in essence, the beginning of this massive neuro net that is really connected. So what if we have one LLM that we're connecting to, or one domain we're connecting to that's not as secure as the others and can take advantage of that? So we, we gotta start thinking about this.

And I don't know if we're going to at the moment, maybe a few years down the road before we get serious about it, uh, when we find out that things are breaking, I'll tell you the part of that that really terrifies me is that we have done a crappy job securing APIs, and all those prompts are going across APIs that are fundamentally insecure. Nevermind the fact that I can hack it in the prompt. So it's gonna be a whole lot of fun.

Steven, you and I have been around this block more times than we care to admit, but are we incapable of learning? It seems like every time that we have a new and emerging technology, we are unable to, uh, process the security implications until, you know, something tragic happens. Well, it does seem like, uh, we, in the name of convenience and, uh, user friendliness have gravitated toward a programming model that fundamentally is insecure.

Because, you know, the, the, the challenge for this, and for SQL injection and many other similar attacks is anytime you have data and system commands mixed together, there's the opportunity to inject system commands into the data. And AI systems are particularly vulnerable to this, because they have no wall at all between the data and the commands. Think about it.

I mean, how, how do you use a chat bot? It's right there in the name. You chat with it, you tell it things, Hey, you know, I want you to add these numbers, or I want you to look for a restaurant in this city.

Uh, that is a mix of commands and data fundamentally. And whenever you mix that, whenever you break those barriers down, there's going to be the opportunity for injection attacks. I mean, that's, as you mentioned, Mike, the origin of so many security vulnerabilities is, uh, people putting, for example, SQL commands into the data side, and then using some technique to, uh, force the SQL system to evaluate those commands rather than evaluating data.

And there's very little you can do to stop this. Honestly, we've been working on it as an industry for decades. Um, you know, the basics are there.

Uh, you know, you want to do input filtering, that sort of thing. You want to evaluate the props. Uh, the other thing that you could do is you could have the machine learning systems, the models themselves be more, uh, resistant to, uh, prompt injection by essentially teaching them what prompt injection is and how to look at it.

Um, I've seen systems where you have two different AI models. One of them, uh, is tasked with cleaning up data, and the other one is tasked with, uh, actually performing the, the task at hand. The problem there is that the first one can be attacked too.

Also, the other, uh, thing I think that a lot of people are leaning toward is, uh, basically non chat type AI applications. Uh, you know, for example, retrieval, augmented generation or rag. Uh, one of the nice things about that is that you can build a system that is somewhat constrained in terms of what data it will process and accept.

The problem is that those are also still susceptible to prompt injection, despite the fact that they are engineered to avoid it. Um, I think we should also point out that the current, uh, topic at hand, the reason we're talking about this today is because this was a company that announced that they could bypass these AI guardrails, and their product is designed to mitigate the attack that they're announcing. So this is in fact, a marketing exercise from a company.

That's the reason we're talking about this. Is that wrong? Not necessarily, because at least we're talking about it, which is a little bit more than we have been recently.

I, I think it's important. I wanna say one thing. Yeah, no, go ahead, Tracy.

I Was just gonna say it's important that what he just said, he, what, what he just said about the fact that we really can't fully eliminate these prompt injections. We can only build smarter, you know, designs. We can build, you know, we can do more training and work better system architecture, but to fully eliminate them, it's, it's sort of like saying, we're gonna fully eliminate open source vulnerabilities.

This is gonna be with us for a while. Well, that that's true. Absolutely.

And I think what you try to do, just like with open source, um, or any anywhere else, is minimize it to an acceptable level. Uh, what I'm concerned about is, um, a relatively out of control market based on foundational models that have been trained on the entire internet, so to speak. I mean, that's, that's, it's kind of funny because, um, we've been talking a lot.

I know, you know, uh, Mike, Tracy, Steven, we've talked a lot about the importance of data in AI and how that has not gotten anywhere near the focus it should have. Um, that includes, obviously, the training data. Most implementations of AI these days are on large language models, generative AI foundational models that are then tuned or adapted or what have you, rather than people training their own original models.

And since all of these foundational models are trained on data that includes stuff like how to hack into nuclear silos or whatever, because it's the whole internet, they just threw everything at them. That fundamental vulnerability won't go away unless those models are retired or replaced with ones that don't have that. And that's just one step.

I don't even know if we'll ever make that step, But having a domain specific model, it makes more sense in instead of having these massive foundational models. And I don't know if we'll make that step either, but now that we have this way to have these protocols and these APIs, we have the opportunity to correct the, the, the course. But these companies who have made so much money off of these foundational models, will they do that?

I don't think they will. No. I mean, they cybersecurity.

Yeah. They, they, they will say in cybersecurity that if you can imagine it, somebody's trying it. And the issue in my mind is that the cybersecurity criminal syndicates out there have infinite time and infinite resources, and are probably training their people on prompt engineering right now as we speak, maybe more aggressively than most corporations are.

And ultimately, they'll build their own AI agent to go kick off a prompt. So I don't have to even know how to build a prompt in the first place. And that AI agent might, for example, will reroute customer service inquiries to some other site somewhere and totally compromise a business process, which to me is a lot more dangerous than just stealing my data and encrypting it.

That's like, you're now, you're taking over my entire business, and that seems a little scary to me, Steven. Is that what we're gonna see? Well, you know, uh, it's a really good thing that we're not talking about laying off government workers and putting AI in it place of them to process all government, uh, actions.

And, and, and I mean, yeah, that's terrifying. And it's frankly true. And like I said, you cannot eliminate this threat.

And so the only way to reduce this threat is to attack it straight on and, and to make sure that you are, you know, doing the things that need to be done. I, you know, I like Tracy's idea of specific, especially fine tuned models. Uh, you know, kind of leaving off of guy's idea, there's also a whole bunch of, uh, people out there that are putting malicious data on the internet in hopes that AI will be trained on it, which is a whole thing.

Whether they are, um, you know, doing it for good like CloudFlare with their, you know, infinite, uh, realms of nonsense that they're directing AI bots to, because the AI bots aren't obeying firewall and robots directives or, uh, for Ill, as guy said, you know, teams of, of black hats who are out there saying, you know, what we're gonna do as bad as we possibly can to mess up this data. Um, you know, ultimately we're going to have to figure out a way to make models that are resistant to this. And whether that means making models that are smaller and more limited or bigger and smarter remains to be seen.

Mm-hmm. I, I think you're spot on in the sense that they should be at least more limited in their, what they're allowed to do, so that I can limit the scope of a breach, right? So I don't have some sort of breach that involves an entire end-to-end process, and at least I can limit the damage in some way or form.

So hopefully that will happen. But, um, guy, I don't know. I think, you know, on the one hand, to Steven's point, it is a marketing exercise by hidden layer.

But at the same time, can you trust the people who build the LLMs to guard their own hen houses? Or do we need somebody else who's a third party to go do this? I don't trust any of these people who built the LLMs.

I don't know about you. Right? I mean, fortunately in AI for years long before these characters came along in the public consciousness, um, there have been the ideas of provenance, visibility, explainability, that sort of thing.

So they, they already existed. And there's been plenty of talk about that. Um, provenance of training data pro.

I mean, this whole, it's just a really complex chain, I guess, or complex is maybe not the right word 'cause it's used too much. Um, how you gain full visibility as a implementer of an AI infused application that uses MCP to co to connect to a lot of other things, including maybe some agents, maybe yours is an agent, um, with insecure APIs and so on and so forth, right? Um, how possible is it to really understand or be able to articulate and monitor, um, where each given prompt and response came from down to the training data and the training data source.

Uh, if new models are created, new foundational general purpose models are created that, um, can claim to have walled out a lot of this dangerous data. Um, and then the entire system from training through inference to, you know, uh, uh, end user result can be audited in that sense. I think conceptually that's possible, but we have a long way to go to get there.

And I'm feeling a little shamefaced at all my enthusiasm over MCP 'cause I love it, without realizing, um, that this sort of inherent danger. And that's why I said at the beginning, I mean, this one, this one deserves all of our attention, um, as people who talk about the industry, because this does feel pretty freaking dangerous. Right.

Bonnie, you've been listening to this conversation for a few minutes now, and I know you're not as security expert, but did you hear enough that might give you cause for pause if you were a business executive that would make you sit down and think about this twice? Yeah, and I, as someone, you know, I used AI a lot and playing around with prompts, I'd be probably thinking of what, what the, um, hackers and attackers are using in terms of their prompt formulation and almost creating scenarios, worst case scenarios of what, what, what they might be doing. Um, just kind of outpace them a bit.

That's just the way I would, I would look at it. But, um, I think it is a big thread, and I think, uh, what Tracy said at the beginning and, and what Steven emphasized as well with the government, it, it is frightening because those kind of, uh, cutbacks might not feel them immediately, but you will, of course, if there is, um, you know, which there likely will be more cyber attacks. But I think AI poses, um, you know, an opportunity to come up with the most, um, cre not really creative, but the, the, and I guess in a negative way of what, what, um, what the security threats are and to try to get ahead of it.

But it's gonna be challenging, as you said, Mike, they're working on this now all the time. Tracy, I have the, I I have a, a, a thought in my head that I'd love to get your reaction to. So will I, in, in the security issues that they generate, be good for application security and API security?

Because once these noise starts to be understood, we'll go address more things that are interrelated. Or will we just focus on AI agents and forget about application security and API security and all that stuff. We'll just go by the wayside.

Well, I kind of feel like a lot of the security is going by the wayside already. I mean, you know, it's always about what the, what the, the kind of the new sexy thing is. Um, but no, we're, we're always, look, according to IBM data breaches is costing globally $10 trillion, $10 trillion, I believe that the private sector is gonna get, maybe we have to hurt more that the private sector is gonna have to get hit more often before they begin taking this more serious.

But $10 trillion, you would think that would make it pretty serious for them, right? Um, I, and I, I just believe that we get so focused on delivering new features out to our, our customers that we forget about protecting them at the same time. And in some ways, we have relied on like, um, uh, cis cisa and n to tell us what we shouldn't be doing.

But when that's gone and we have to start figuring this stuff out on our own, um, the, the private sector's gonna have to get, get busy on it. And I don't think, I think that they're going to step up to the plate, especially if that number doubles if we're looking at $20 trillion, because now we have not just SQL injections, but prompt injections, as you pointed out, Mike could take over more than just, uh, the data, but their business itself. All right, Tracy's on record.

20 trillion is the numbers. What's, what's your number for when we get enough pain and people actually start doing stuff about this? Uh, is never an answer.

Oh, gosh, it's a legitimate answer. I Can't believe you, I can't believe you bypassed that opportunity, Steven. I'm gonna say rigsby sounds cynical.

$1, $1 trillion. Yeah, I, I think we're there. Um, Tracy, I, I completely feel your, I mean, I, I, you're absolutely right.

Not enough attention is paid today to security. Um, but I, I actually think that over a 10 year ramp, AI aside, there seems to be a little bit better and closer attention to security. I don't know if we're on the upward curve of the hockey stick.

Finally, um, AI would, and these issues are just gonna accelerate it. So, so $10 trillion seems to be the number. It just depends on how much money we have, right?

If we hit a recession, the things that fall are security and training. Security and training. That's, and education, security, tra uh, training and testing.

Those are the bottom of the kind of the, that's the last thing that we like to, to deal with. And we put money into it. When there's money available, we hit a recession, we're not gonna be doing a lot of work in this area.

Companies are not gonna be investing in these areas. There is a sports book in Vegas somewhere that somebody's creating that's gonna have a bet on what is the top only the cost before we finally get, take this seriously. So there you go, folks.

We'll be back in a minute to talk a little bit more about ai, but this time, AI infrastructure. Stay tuned. Hey, folks, we're back.

And we're gonna talk about, well, the infrastructure that makes all this AI stuff works, because our friends over at Tech Deal, they had a, was that a four day event on AI infrastructure? Is is that a record for you guys? That's not the first time we've ever done that, but, uh, it's pretty rare that we do it, and it shows just how much interest there is.

All right. And we previewed this event last week, but, and there was the actual event and some lessons learned, and I know that guy was a delegate there as well. But Steven, walk us through the high points a little bit in terms of what happened and did anything surprise you?

Yeah. Um, I would say that the thing that was most surprising in a positive way was how so many of the companies, and hopefully Guy saw this as well, so how many of the companies led with, uh, customers and results rather than infrastructure. You know, you would think that an event that features companies doing some pretty low level stuff with storage and networking and compute would, um, you know, really kind of miss the forest for the trees.

But, uh, the complete opposite was evident. Uh, you know, companies again, and again, and again, we heard this from Juniper. We heard this, uh, from soy, we even heard this from some newcomers to the event like Fon, who we haven't ever seen before, uh, coming in and, and really trying to focus on what is the usage, you know, what is the value of this infrastructure technology rather than what is the, um, just the product like kind of mu nuts and bolts?

What does it do? Uh, did, did you hear that guy? Am I, am I off base?

No, no, you're absolutely right. Um, there, there were definitely instances of, um, this widget is bigger than they used to be, um, and therefore you should love it. Um, but, um, I, they, there seemed to be this overall bent, and I loved seeing it from Fon.

Well, I wasn't there for Fon, just to be clear, but, you know, um, I was there watching it virtually, like everyone should. Um, so, but from Fon, Google Cloud also, which was, uh, uh, an all day of presentations that I attended, such a focus on the really interesting challenges. And I say interesting for a reason.

I think, you know why I say interesting, very may we Live very interesting times. Yes. Yeah.

Well, also, we did a preview podcast that, uh, uh, on the first day that was, that said, should AI be boring? Really interesting challenges, um, having to do with, uh, you know, very wide ranges of usage patterns, various type, many types of AI services to be included, multiple AI services, all of these things that create challenges to traditional infrastructure. And it's not just about throwing more and more horsepower as in the form of GPUs or anything else.

And so the networking is important. It was great to see the networking presence, whether it's Juniper or Google Cloud had, uh, two separate sessions during that day on networking, um, data. We were just talking about it, the importance of data.

You have solid, I there fis on there. Um, so yeah, all of these addressing, um, how difficult it is to design, build, and operate, um, for better ai, um, both in terms of efficiency and in terms of responsiveness, effectiveness and everything else. It was very much of a customer user and operator focus.

Steven, I'm not sure people get this just yet, but it seems to me anyway that all these AI applications run fundamentally differently than the previous generation. There's a lot more data moving around, and I don't think the existing infrastructure we have in place is designed for this. So are we gonna wind up basically replacing our existing infrastructure to run all these AI capabilities that will be embedded in just about everything we're gonna run?

Well, yeah. Let's zoom in on that for a second. As guy said, um, we heard a lot about networking.

And in fact, in the later days, uh, we, as you said, we, we had Juniper, um, you know, Google brought up networking. Uh, we also had other networking companies as well, including, uh, net and VI and Keysight, uh, all of them. If, if I was to pick out a theme from the networking companies, and, and honestly, this is the same theme from the storage companies as well, is that this AI application, whether it's training or inferencing, you know, running things, uh, in p in production, it needs different, uh, network infrastructure, different, different connectivity.

It has different access patterns, and all of these companies are really working hard to basically redo what they've already got. Now, they're not trying to reinvent the wheel. In fact, I have, I have to say, I'm pleased to see the wheel, uh, coming out strong.

If you know, what, if you, if you catch my metaphor, I mean, we're hearing a lot about ethernet, we're hearing a lot about RDMA, we're hearing a lot about the, the, the traditional software defined networking, and that's these traditional topics. But what they found is that these same techniques can be used to build networks that are more appropriate for the unique challenges of ai. You know, on training, that means interconnecting tons and tons of GPUs, and, uh, you know, Google talked to how they're doing that internally.

Juniper talked about how they're building that with companies like nvidia. Um, you know, Avis and Net talked about how they're integrating and, and making networks smarter and more responsive so that they can support this really crushing, uh, new network topology where you have, you know, this incredible array of, of, of low latency, high performance consumers all clustered together, all ne next to each other, all demanding access to just a ton of data. And then the final one that I want to call out was Keysight.

Their whole, uh, product, their whole mission is basically to prove that this thing is gonna work before you spend a billion dollars building it out. And that can be a pretty challenging proposition, again, because these workloads are just so different. Mm-hmm.

Tracy, do developers need to know more about infrastructure these days than they did in the past? Because at least as far as I can tell, there isn't some magic compiler out there that, you know, strings all this stuff together for me. So is it getting harder to build apps or easier?

Uh, you know, I believe that every developer should have taken a course to learn to write their own compiler or build their own infrastructure because it's so fundamental in, um, understanding how to write good code. Um, so yes, 100%, uh, there needs to be more knowledge about how, uh, you know, what does an AI in, what does ai, AI infrastructure actually look like? What are the problems that we're facing?

The developers need to understand that because they're writing to deliver to it. Not only that, uh, when you really understand the infrastructure, you start asking questions like, maybe we should have multiple domain LLMs instead of these fundamental, uh, large domains that we've been using, because then they begin understanding the security implications of that. So when we're just throwing, you know, when we just go, okay, there's a, there's a model out there and I'm gonna use it, we're gonna start writing against it.

It, it does lack some depth, right? You're, you're, the end result is not as good as it could be if the developers had a better understanding of it. So I hope more developers are attending these, um, kinds of conferences.

I would love to, I don't have the time to. Um, and I always wonder, is this gonna set us up for Quantum, because it's just around the corner, right? Is does this help us with quantum getting to quantum?

There is a lot of talk about quantum, especially on the securities focus. I mean, we heard that from HPE with the, you know, the ProLiant becoming quantum ready. Uh, we also did hear that from, uh, basically everyone who talked about encryption was concerned about, you know, quantum ready encryption.

Also, Tracy, I'd love to have you at Tech Field Day sometime we'll get, we'll get to there, but it'd be great. So I have it then. How, how far are we realistically for a, a private company, a large private enterprise, to have their own AI data center?

Um, I think that the point that was made by some of these companies is we are there, um, basically there, you know, Nutanix, uh, was in there talking about how you can basically just kind of point, click and boom, you got yourself, uh, an AI data center, AI infrastructure, all completely baked, ready to go. And crucially, to your point, supporting multiple different models. Uh, you know, you don't have to say that it's gonna be these, this one model, uh, frankly, even Google who has their own AI models was talking a lot about supporting other people's models and, and giving customers the choice of what they wanna run, where they wanna run it.

I think that's good. I, I do think that we are already at the point though, where an enterprise, if they chose not to run in the cloud, could literally just point and click their way to having, uh, AI applications. And Then I have one.

Absolutely. Those are, those are, I have one other question. I'm just gonna say that's, that's inference, that's largely inference not training, but Yeah, go ahead, Tracy.

You want one? We'll, we'll, we'll send you one. So, okay.

So I, I'm imagining, I, I know maybe I'm getting too far ahead of us, but, um, if these companies are gonna start building their own AI data centers, how important then is our infrastructure and the, the availability of chips, we are faced with a problem in both those areas? So is our, is our geopolitical situation going to stop us in our tracks, or are we solving that? Well, The, our geopolitical situation stop us in our tracks, uh, along many frontiers.

Um, as you know, um, this one included, um, the, the importance of understanding that, that we're talking about, um, ai, private AI data centers for inference across multiple models and all that sort of thing, and some tuning and rag as well. Um, is that, um, a lot of those applications don't require the famous by now famous GPU. Uh, there are a lot of, there's a intense development, um, especially in the ARM community, but also in X 86 around, uh, AI inference that's CPU only.

But in fact, there's also a whole lot of, uh, development of non G-P-U-C-P-U helper processing units like the data processing unit, DPU, like it neuro processing unit NPUI hesitate 'cause I feel like it's a kind of a specialized GPU, but then there's a tensor processing unit that, uh, that Google has, has created. Um, what, what, what was that system? Uh, Google, uh, talked about, it's like a 43 exo flop system.

And when, you know, an old CPU guy like myself sees that and knows that the biggest supercomputer in the world right now barely hits two exo flops. He kind of scratches his head to use Mike's phrase that I always use every time I come on this show. Um, but that 43 exo flops comes from using tensor processing units.

There's more, there's asics, there's all kinds of ways that the CPUs duties, especially operational and managerial and orchestration duties get offloaded onto a specialized processing unit of one form or another. And then the CPU can concentrate more on things like inference training, different bag, especially foundational training. But yeah, you can have the, there's a, a, you know, an ai, a kind of an AI data center in a box, quote unquote, that Google has just announced it's a mini sled that probably would fit in your office, Tracy, if I'm seeing right.

Um, if you have the power, and that'll do inference right there for you. So a couple of things on that part. One, HP, Dell, and those guys will guarantee that they will get you the GPUs that you want because they have deals with nvidia.

So it's not like I gotta call up Nvidia directly be though, be wary of benchmarks. Um, most of the guys that I talk to who are building AI apps, uh, don't really take anything other than the gpu seriously, because, um, those benchmarks don't reflect the, uh, degrading of the accuracy of their models running on something other than a GPU. So a lot of them are kind of saying, yeah, it's nice to see that there's alternative processors out there, but, uh, from their, from their nickel, they're like, eh, I don't think we're going that route because I'd rather have an a less expensive GPU than rely on some other architecture that's not really optimized yet for processing inference workloads as much as those benchmarks might suggest.

But Steven, I don't know what you're hearing, but that's what I get. Yeah, I, I would say that, uh, when you say benchmarks, that could encompass a bunch of different things. I think that, uh, absolutely the rough numbers, the sort of, you know, we can do this many exa flops.

Yeah, that's just a bunch of hoopla. Um, yes, theoretically you could do that many exa flops, but on the plus side, I am seeing a lot more awareness of the point that you're making, which is the only thing that matters is what you're actually able to do in the real world. And so, for example, the folks at ML Commons, uh, I know that they've drawn some flack for their, uh, some of their earlier benchmarks, but they are keenly aware of this, and they're really focused on trying to show real world performance and not just at the top end.

I mean, they've got client, uh, benchmarks, they've got storage benchmarks, that sort of thing now. And I do believe that they're working at least toward having things that are more reflective of real world use case. But we also heard as well, um, to Tracy's point that, uh, and this kind of rhymes there, that, um, the challenges that developers are developing with an expectation of a level of performance that may or may not be able to be met on different platforms.

You know, if, if all your developers are developing toward cuda, then it really is not practical, even if it, even if you can run it somewhere else, it's not practical to run it somewhere else. And similarly, if they, uh, have an expectation that we'll have this much storage or networking or processor floating point performance that may not be applicable, whether when you port that to arm or to risk five or to a tensor processor or something. So, so really what we're finding is you need to have this sort of holistic picture, the entire infrastructure all the way from top to bottom, all the way from API and, uh, platform all the way down to storage and networks.

And ultimately, that's the only way to know that something's working, which is why some of these pre-baked platforms, like you mentioned, from companies like HPE, uh, as guy mentioned from companies like Google or Nutanix and, and some of these others, that can make a lot of sense because essentially they're not delivering infrastructure so much as they're delivering a proven platform that can run this or this or this at this rate. And, and maybe that addresses some of this concern. Maybe ultimately though, I think organizations need to figure out how they wanna structure these teams, because, you know, when I talk to folks, there's a debate going on.

Some of them are saying, you know, the IT ops team is gonna take over a, uh, AI inference and manage it just like anything else. Other folks are saying, we're adding IT ops folks to our data science teams, and those guys will run this whole thing. I don't think there's a right answer there, but one thing is for certain folks, infrastructure you have today isn't gonna hold up for tomorrow.

We'll be back in a minute. Discover techron group, the epicenter of tech innovation. We are your go-to for reaching IT leaders and practitioners worldwide.

Our secret impactful content that sparks awareness, engagement, and top quality leads with us. You'll access editorial websites, streaming videos, virtual events, custom content analyst research, and more. Join our satisfied clients.

Let's revolutionize your tech journey. Contact us today and tell your story to the world in the most powerful way with Textron Group. Welcome back to the Techron Gang.

Well, since this day and this week is all focused on security, I decided to talk about sustainability and cybersecurity and perfect timing. Hitachi Vanta, VSP one, uh, really does combine both features in their platform. So let's take a look at a video I did, kind of explaining it, and the features and its connections to ESG, Cybersecurity and sustainability.

Both are critical, urgent, and now they're converging across enterprise IT infrastructure. Hitachi Van Tara recently introduced its virtual storage platform, one or VS P one. This data management system is designed to meet these dual demands.

Its automation supports compliance with Europe's corporate sustainability reporting directive, giving users better visibility to reduce carbon emission on the security side vs. P one pairs unchangeable snapshots with AI driven ransomware detection. There are some challenges though actual energy savings depends on variables like workload type, facility design, and the local energy mix.

Plus stronger cybersecurity can require more processing power with AI workloads rapidly expanding, even the most efficient systems face pressure to keep consumption in check meeting, both cybersecurity and sustainability goals will require ongoing adjustment as threats and regulations evolve. So this platform is, uh, built to meet energy star standards. And another interesting factor about it is it's also designed to have a longer life and be more upgradable than previous, um, additions.

And it's 40% more sustainable. That's, that's the, the claim of it. So it has a lot of different features, AI driven, more efficient, and an an autonomy where it knows when to switch, um, energy sources.

It knows when to, um, look, become more aware of carbon emissions as it goes. So an interesting feature, and I thought it would be a timely thing to talk about today. So all of these platforms are generating now sustainability data reports, and it's another flavor of the telemetry data that the machines kick off.

But, um, Steven, is it your sense that maybe cybersecurity criminals are gonna go after that data too, because well, I can, I can learn a few things about your data center environment pro, Well, certainly cyber criminals are already going after storage platforms. That's something that we've seen quite a lot of, where they have actually pretty good level of understanding of the fundamental, uh, storage platforms that are being used. And they're starting to use those as ways of basically counteracting the, uh, ransomware readiness that some of these platforms have, which has led to a bit of an arms race.

So it's nice to see Hitachi taking this seriously. Those of you aren't familiar with Hitachi. They're, uh, one of the leading, um, providers of real enterprise grade storage, essentially the big, big iron, the stuff you see in the data center in the movies, and you may not see it, um, in smaller companies.

It may not be a, a, a household name, but they are, you know, kind of, you know, one of those, the IBM of storage, that kind of thing, along with companies like Dell and NetApp, pure Storage, and now Vast. And all of these companies are trying to basically evolve their systems to be more, uh, cyber ready, but also to be more green. To Bonnie's point, uh, it's good.

I'd say this is motherhood and apple pie when I hear about a major company with their main major platform, um, adding things like cyber readiness and, and better, um, you know, green, uh, technology, uh, you know, kind of like when Mercedes added an electric version of the s class sedan. Yeah, yeah, let's do this. But at the same time, uh, there's also the question of trickle down.

Uh, will this be applicable to the rest of the product range? Will this get out into the broader market, and are we actually just opening up more doors to, as you say, to cyber criminals who might use these reports, these features as a way to cause problems instead of solving problems? I, I guess it means you have to have a overall perspective of, of what the whole system is doing and, and, and have decent, um, operational and and management capabilities to keep that from happening.

Uh, Tracy, on the security side, are, are companies, uh, talking about the vulnerability of storage systems? Now I have, that is a topic I have never heard, but that, you know, security is such a broad area and I focus so much on, uh, the supply chain that's coming through. Um, no, I have, I have not heard that.

Um, but you know, I wonder with topics like this, it's great that we're seeing more efficient CPUs, but we just got done talking about NPUs and GPUs. Uh, so is this technology going to be relevant for NPUs and GPUs in the future? And certainly CPUs is a good place to start because most everybody who's running a machine at home, I never turn off my computer, never.

I don't even think about it. I just, I just walk away. I might turn off my monitor, but I don't turn off my computer, so yay that that's happening.

But how does that apply to the next generation of, of computers that we're building, Guy, do you think people will replace existing storage systems because of security and compliance concerns, or is it more likely that, you know, as I decide that I need better performance or increased capacity, that then I look at security and compliance as kind of a checkbox on top of that. But it's, um, I mean, I'm definitely hearing a attacks The witness. Like, I mean, I'm, I definitely hear about a tax directed at storage systems, especially ransomware, but I don't see anybody doing anything about it per se.

Listen, um, the tape drive market is still a healthy, and maybe not as, maybe not particularly large, but a healthy and, and, and within its realm thriving market, um, now that would be selling new tape systems. But why are you selling new tape systems to replace and consolidate old tape systems, uh, storage. And to my mind, memory and cache have become this big continuum from the most immediate, most needed, all the way down to the coldest.

And the ones that you need to be able to stand and look and say, Hey, see, here's the data here, type storage systems on-prem and everything. I think that that introduces security vulnerabilities as these, this, you know, I call it a continuum. It isn't really, but a system connects to system and in more and more automated ways, um, sometimes I think maybe you like to call on me with questions so you get the cynical and blunt answer.

I think the cynical and blunt answer is, these old systems are gonna stick around and there's only more data coming in. Transferring data is, Steven knows more in his little finger than I know in my entire body about this sort of thing. But I think he would agree, yes, I think he would agree that, um, transferring data, even within a data center from one major system to another can be a considerable difficult time consuming and error prone process with all the automation and sophisticated systems we have now.

Um, let alone moving it to another data center, moving it into the cloud, moving it out of the cloud data is probably the stickiest of sticky applications, if you like, of, you know, out of, out of all the infrastructure. So you didn't have to lead the witness. Mike.

I, I think that you're right. Um, there is a whole lot of opportunity, let's say, to layer security and compliance on environmental, everything else on top of existing storage systems. Not, well, just please tell me they're already doing that.

I don't wanna think that, I mean, aren't they at least taking some kind of, you know, like snapshots and being able to do rapid recovery on the data let's, and secure access, of course. So I, I feel like they've, we've made, they've probably have made more progress since The software supply chain problem. Yeah, and I do wanna add one thing, which is much like security, I think the cloud providers are doing a phenomenal job much better than almost any other actor in this whole, um, uh, uh, system.

It's just, again, the expense of getting the data in the expense of getting the data out, that's, that's the problem with cloud. But once it's in there, it's, it's, it's pretty good. I mean, Steven, am I wrong?

Yeah, just, just to level set to Tracy's point, um, yeah, storage has always been sort of a weird little corner of it. And I will be the first to admit, I'm Mr. Stor.

I wrote a book on storage, wrote, uh, lemme tell you, it is not all that secure. Historically, security has not been considered all that important in the storage industry. But that being said, the concept of snapshots, of immutable snapshots, of cloning, of data replication, these were the key features all the way back in the 1990s with enterprise storage.

And they remain some of the key features today. And now to the point that you're talking about the, the addressing of this data gravity problem, the fact that it's difficult to move storage, uh, at all, let alone, to move it in a way that's useful is, uh, the thing that a lot of these companies are addressing. So when you look at these next generation products, including the VS P one, which is what Bonnie was talking about in this video, one of the big features that they're adding is essentially making your data appear to be anywhere, all everywhere, all at once, uh, to quote a movie.

Um, and, and that's good, but it means that security is even more important because frankly, if you can access it seamlessly, you know, your enterprise data seamlessly in the cloud or seamlessly at the edge or in, in a, in a AI environment or something, well then you really need to make sure that the systems that are protecting that, including access controls, but also to Tracy's point, things like snapshots and clones and, and immutability. You have to make sure that that stuff is up to snuff. So that's why I'm really happy to hear the fact that all of these companies are working on that.

And I can say that with confidence, every enterprise storage company is keenly aware of the challenges of access in this sort of hybrid cloud world that we live in. And also the fact that there's a lot of ransomware and, um, you know, ex data exposure and data leaks and that sort of things that we have to be aware of. So even as they're building these systems to be bigger, bad or stronger faster, they're also trying to address the cybersecurity issues.

And it's also exciting, as Bonnie said, that they're working on addressing the, the green aspect, because that has never been a priority at all. You run these things at full tilt and who cares how much power they're using? Well, the modern world needs that too.

And so again, it's important to see the companies addressing, uh, All right, well, I'm gonna say this and close this out from here, but, uh, honestly, more than 30 years ago, the, one of the first things I ever learned about it was, nothing good happens when you move data. And here we are still talking about it three plus decades later. So that was a wise Person who told you that there's gonna be a lot more data to, to protect and move around and store with ai.

That's gonna be massive. Hey, Tracy, can I, I know you're closing that mic, but Tracy are, are, are developers thinking about moving to the data? I mean, is that a paradigm?

I don't think so. No. Yeah, I didn't think so either.

No, I would not think that. All right. But just like the economy, it's all about the data at the end of the day.

So folks think twice about that data, where it is and how it's stored because the Batten guys are after it. I'm just say, Hey, wanna thank Anything? I'm just say it's all about the chips.

We got a problem with chips. That was last week, Sean. This week's about day, It's about to reinforce it every single time.

It is all about the chips. That's the fundamental part of this puzzle. Alright, sounds good.

Hey everybody, thanks for sharing your insights today. And thank you all for watching the latest episode of Techstrong Gain. You can find this and other exciting content coming up right behind us on Techstrong tv.

Please stay tuned. Until then, we'll see you tomorrow.