Red Hat Lightwell Targets the AI Vulnerability Patch Crisis

Hey, guys. Thanks for the throw. We're here with Brian Gracely, who's senior director of portfolio strategy for Red Hat, and we're having a little chat about this new initiative that they have launched to help with this vulnerability crisis, I guess is what I'm going to call it, because we have all these AI models that are pretty advanced, and they're bringing home to roost this massive amount of technical debt that maybe we've been ignoring for many, many years.

And Red Hat has a clearinghouse initiative to help deal with some of that, especially around open source code. Brian, welcome to the show. Hey, Mike.

Great to be back with you and good to talk to you again. So explain to us exactly what it is you guys are up to and how this whole process comes together and who's involved. Yeah.

So, obviously the world got kind of a shock, a scare, maybe back in April when Anthropic released the Mythos model and started to put out some numbers about how they were digging into all sorts of critical software. And we had been kind of talking about this space for a while prior to it coming out, but that was sort of the wake-up moment for the industry. '" And then they started kind of looking at, okay, we're not going to have the resources to fix everything in the world.

The world is powered by software in all sorts of ways. Where would we focus our efforts? And the message that we got loud and clear from a lot of these large organizations was they were as much concerned about obviously the new things that they have, but they were very concerned about the systems that they have that are a little bit older, have some technical debt, but basically run their business.

And they sort of said, "Look, we know we've been neglecting them for a while. Upgrading them and constantly upgrading them probably isn't a realistic thing. So, if somebody were to offer the ability to take our vulnerabilities, we throw a Mythos at our existing systems, our three, five-year-old Java systems and Python systems.

If we were to give you all those vulnerabilities and you guys patched them for us, specifically where we are today, so I don't have to go through a reassessment of my thing, I don't have to recertify things, I don't have to upgrade. " And that's ultimately what Red Hat and IBM pulled together for the Lightwell. Originally, we called it Project Lightwell.

It's now formalized as Lightwell as a commercial offering. But it's very focused on using these new tools and technologies, and really trying to stabilize the systems that the customers rely so much on these days. So when you think about all of that, what makes a clearinghouse the right approach?

Because, I don't know, I'm sure you've seen it, but a million people announce some sort of initiative to go address this issue, but it's unclear what's different between one and the next. Right. I think we always kind of go back to certain principles that Red Hat has seen over the last 20 or 30 years that work at scale.

And so the question that everybody's really asking is not like, is a clearing house the right model or some other model the right model? It ultimately boils down to, how do we address this problem with some sort of scale so that it's not individual actors burning through their AI tokens to fix it once, and then it doesn't get propagated in any way, shape, or form? And what we sort of came up with was we wanted to blend the best of how open source works, which is anybody can contribute to the solving the problem or identifying the problem.

You bring a collective set of experts together, kind of in the middle, if you will, similar to what an open source maintainer does. And then, as things are fixed, you push those back to everybody that has an interest in doing it. And so the clearing house model for us allows us to work with customers who are interested in bringing their challenges, their vulnerabilities, their old CVEs to us.

We go about fixing them, but because so many enterprises have an 80%, an 85%, it's pretty common amongst them. Unique for an industry, but a lot of it's common. A fix that one Lightwell member might bring to the group is probably going to be relevant to dozens or hundreds of them.

So we get that scaling thing. And then the second thing that we do is it would be one thing if we just did this as a commercial entity. " The second piece of it that we do, that obviously, again, plays to Red Hat's expertise and DNA is everything that we fix, after we've given it to the members under embargo so they can fix it ahead of the attackers, we're going to give those back to the open source community.

So we're not only going to patch that older version, we're going to patch the new version for the upstream. And that both helps the overall project, but it also relieves a lot of the pressure that open source maintainers are dealing with right now, just with the amount of AI vulnerabilities that are being thrown at them. So we're trying to be both good Good partners for our customers, but also good citizens for the open source communities that need help.

And in that model, therefore, the maintainers who often complain that they don't get any help from the people who are consuming their software actually start to see something that feels like a benefit in participation from people who are now helping them to fix some of these vulnerabilities, correct? Yeah, we hope so. The feedback we've gotten from most of the communities that we talk to, and we've been participants in those communities for a long time, so Java and Python and others.

Yeah, they're looking for both sort of help from a volume perspective, but they also want to maintain a certain amount of control. So, we've always said, "Look, we're going to provide back to you what we've fixed, what we've verified, and so forth. It will be tested.

It will have an SBOM with it. " But they still have complete control over if they want to accept it, if they look at it and make a change to the process, maybe how it got fixed. So we're still absolutely giving the open source communities the control that they want to have.

But yeah, we're trying to relieve some of that pressure they get, and we think over time, again, looking at places where we can scale, that scale, again, helps downstream as well. How fast are these patches and updates being applied by the end customer? And I ask the question because historically, they were slow to patch things- Sure ...

and they tended to wanted to wait to see if the patch was stable and it wouldn't break their systems. And so they all kind of just kept an eye out to hear what other people were doing. But it seems like the adversaries now are creating exploits even before the patch is out.

So has that- Right ... changed the pace at which all this needs to occur? Yeah.

And I'm happy to pass it along. There's some recent analysis and data that's come out that, yeah, the gap between finding a vulnerability and it's being exploited is getting down to almost zero. " And they see the benefit of that.

" So it's taking one problem, this Mythos sort of threat that they have, it does turn it into two things they have to think about. One is, how do I get the patch? We think we're helping with that.

And then the second piece is what I call sort of patch to production. How do you get that today if it's taking you 60 days, 90 days, whatever? And in a lot of cases, while they don't love that number at first glance because they go, "It's shorter than we typically do," they also realize that if we made the number 15 or 21 or whatever, the attackers are going to get access to that information, whether they got it upstream from an open source community, because they're watching those communities.

They're keeping track of what's going on. " The good news is it's partially a technology thing. There are some technologies that are going to help them get through CI/CD better.

They can leverage AI to build test coverage. There is tons of automation available to help them deploy quickly and in batches. And some of it is them having to make some tough decisions about, do I need to wait a week to have that sort of review meeting before we move something out?

So it is something they're aware of. It's one of those things where they're just sort of going, "Look, this is the new world that we live in. " And we're beginning to see them put the things in place so that those patches don't just sit on your countertop like bananas that you don't eat, and they go bad.

That's the thing we don't want to have happen ultimately. We've also historically counted on vulnerability ratings to kind of determine what to prioritize, but- Yep ... as I look at some of these new attack vectors that AI is servicing, they seem to be now also able to chain together various low-level vulnerabilities to create something that is more lethal.

So do we need a new approach to thinking through how we determine what vulnerabilities need to be remediated first? Well, yeah, I think that's the other big sort of aha moment. " I think the other piece of it is what we're dealing with is not only a bigger volume of vulnerabilities, it's also the velocity of those vulnerabilities.

" And then, the bottom two-thirds that were yellows and variations of that would kind of get pushed down the road. And because a lot of those were, well, those two low ones aren't going to ever get linked together. Humans never going to link those together and create a high severity out of those couple of them.

" They got stuck, and in the old days, a human or a deterministic sort of script would just quit. And nowadays, these agents, as long as they don't run out of tokens, is going to keep trying every mechanism they have possible to go try and exploit stuff. And so we see them going into adjacent networks, going into servers that they never would've had visibility to before, or something along those lines.

And so, yeah, it kind of goes hand-in-hand with the thing where if a customer said, "Well, I'm just not going to fix it in 15 days," because we just don't do that for a lot of them. " So yeah, those two things have been very much eye-openers and I think for the customers that have experienced a Mythos or any similar kind of tool, those have been the aha moments that until you kind of experience it, you don't really realize how potentially challenging the situation's going to be. Of course, Mythos was kind of the wake-up call, but it also seems to me the reality of the situation is, many of the readily available AI models are already pretty adept at discovering vulnerabilities, and the open source community seems to be getting better at discovering vulnerabilities.

So how much time do we really have before this is a full-blown crisis versus one where we kind of see coming now but hasn't quite gotten here yet? I think everybody who's tried to predict what AI is going to look like three months, six months, nine months out has been, in some cases, an order of magnitude slow in what they think is going to happen. So, I think the reality that people have to deal with is, Mythos came out in April.

It was a very small number of people. It's since expanded, so we're seeing more people that have sort of opened Pandora's box and said, "Oh, wow, okay. " Obviously, we've seen other tools come out, GPT Cyber, and we'll see open source versions come out.

I think it's going to be, in the second half of this year, it's going to be a pretty widespread reality check for a lot of people. And then it's going to be a very interesting thing to how quickly companies can kind of recognize that this is the new normal, and start to take action about that. Because I'll throw out an anecdote without sort of naming names.

" Maybe more so than they had been in the past. So, it's worth kind of getting past the headlines, experimenting with the tools, and sort of beginning to understand, okay, how is this going to impact us? Do you think we'll see more regulations show up that have specific requirements for how quickly you need to patch something?

Is that the next logical step here? I think the first thing that we're seeing, we are definitely seeing a lot of organizations that have regulatory, at least, scope, start to mobilize. They're trying to figure out, okay, what do we need to do?

I think the first thing they're trying to figure out is at least communication and notification, right? " It could be your region, it could be your industry. That's the first thing that we're seeing all over the place is this, I have to at least notify people that I'm aware of the problem.

And then, yeah, I think to a certain extent, we'll see regulation try and put some things in place. But the reality is, regulation is typically a year to 18 months behind what the leading edge of anything is. So, waiting for regulation to come along and sort of save the day probably isn't a good strategy.

I think being actively involved in how you're going to address this is the best first step, and then as regulation comes along, or probably more importantly, as community comes along to give you scale, that's the next thing to get associated with. I think it was Winston Churchill who said something to the effect that we should never let a good crisis go to waste. So, will we, at the end of all this, wind up with maybe much better DevSecOps workflows and maybe even more secure applications?

It might be painful, but is there more of an upside than we appreciate? Yeah. Look, at the end of the day, put all the scary stuff aside, the anthropic, whatever you think about that stuff.

The industry as a whole has been talking about this stuff for six, seven, eight years, right? In the context of not even so much security, but we've gone through this last decade of kind of digital transformation, which was, how do I take processes in my business or differentiation in my business, enable that through software? And what we've seen for a lot of companies is the companies who ultimately manage software better than other companies, whether that's putting new features out or leveraging data better, or using it to reduce cost or create a differentiation, those companies tend to win.

And so this is really an extension of the better you manage software and rapidly changing and moving software, the more prepared you're going to be in a dynamic market. And so, yes, this gives us a distinct use case to talk about it. The good news is, especially on the patch to production side of things, a lot of the fundamental technologies are there and available to people.

It's just a matter of sort of adopting them and bringing them into your culture. So, yeah, I think this very much could lead us to, we'll look back on this a year from now and go, whether we fixed all the vulnerabilities, probably not, but do we have better systems for managing software quickly? I hope the answer to that is yes.

All right. Well, folks, you heard it here. There's going to be pain.

There's just no two ways about it. However, the good news is everybody's experiencing the same thing at the same time, so maybe we can all work together and make it less painful for all concerned. Absolutely.

Hey, Brian, thanks for being on the show. Thanks, Mike. Appreciate you having me.

All right. And back to you guys in the studio.