Addressing Advanced Threats in Open Source with Sonatype’s Brian Fox at OSS Seattle 2024

This is Textron tv. Hey everybody. Mitch Ashley here.

We are at Open Source Summit in Seattle. A great conference, uh, day two for us, and we're talking to a lot of fantastic people. The great thing about Open Source Summit, something that, uh, Brian Fox and I, uh, who's with the son and I we're talking about the diversity of projects that you get to experience and, and see here versus a security conference or this kind of Kubernetes or whatever it might be.

So good to be chatting with you, Brian. Yeah, likewise. You know, there's, uh, we were talking earlier about, uh, the, the people might notice the United Healthcare, um, or the, uh, xd, sorry, excuse me.

Um, attack that happened a lot of fascinating things about it in a security sense. One of them for me was the impact of the ecosystem. It just wasn't.

United Healthcare is packed all their Yeah. People that they had to pay for, pay their providers for, but it also was an example of a source project where some got maintain their status over a long period of time Yeah. To build up that trust Yeah.

And do that. The, the XC one was actually a, i I separate, but what they do have in common are sophisticated possibly nation state attacks. Right?

Mm. The healthcare one, united healthcare situation, uh, I forget the name of the project that That one was. Oh, it was, it was two different lists.

Okay. I'm conflating two. Yeah.

There, there were two different ones. But the, but, but equally scary and sophisticated, you know, the United healthcare situation, being offline with payers not being able to, you know, or, or billers not being able to get paid was Right. Is a, and, and, and as far as I know, that situation is still going on Mm.

Uh, months later. But the XZ thing is something that came out, uh, came to light just before Easter. Okay.

Which was, um, you know, basically XZ is a compression library that's used, uh, on Linux, uh, underneath most of the distribution. So the, the scope is quite large. And what came to light was that there was, um, an attempt, not an attempt, an actual takeover on the project all the way back in 21 that, uh, uh, uh, a new person showed up on the scene.

There were a number of emails from what we believe were sock puppet accounts using the same person Mm-Hmm. Using different emails to try to put pressure on a single maintainer project to get help, which is the kind of thing that when you're a maintainer on a project and you're kind of burned out and somebody comes along and wants to help you, that's that's what you're looking for, right? Yeah.

Yeah. And so they played right into that. And then, and then over the next two years, systematically included pieces of a, of a puzzle that added up to basically an attempt to put a master key into, um, SSH, which, if it got distributed on Linux systems, that that would have massive impact, right?

Mm-Hmm. And, and the level of sophistication of this attack combined with how long it played out is a dramatic escalation in, in, you know, what we're seeing in open source. Yeah.

In the security world, they talk about advanced threat prevention, which is a slow roll attack, right? Yeah. It isn't a, you know, quick bombardment of attacks against some target.

That's right. It's that over time, I, I was using the term this kind of the sleeper cell Yeah. Example where it's been there for some period of time That, that, that is the perfect model.

I think that's what we're seeing. And that's, that's the scary part. And so, you know, like we were talking about before, you know, when I saw this, the first thing that came to my mind was that the, the, the chances are that these attackers collected one project two years ago, and we're content with that, um, are like zero mm.

Right. Why wouldn't they go and try to do the same thing on more projects? And so that's, that's where I've been looking, trying to analyze all the signals that we have to see if there's anything that we could, you know, uncover.

Um, some folks approached me a couple weeks ago with another situation that, you know, the details of, which haven't been, uh, publicly released yet, but there was a press release to open js and the open SSF foundation put out earlier this week that basically outlines, um, the fact that we've now seen the same pattern in another project. This is not a copycat, because it happened before XZ came to life. Okay.

Right. And so the shape of the attempt with, you know, multiple, uh, uh, sock puppet emails, pulling on the heartstrings of a, of a project, trying to suggest that they let more maintainers come in, um, are identical. Right.

And so, sort of confirmed the thing that I already in my heart, believed to be true. Mm-Hmm. Um, and so that doesn't make me feel better.

I was gonna say afraid that you're right. Afraid that I'm right. And it, and it shows that that's actually the situation.

So I think, um, we're looking at the tip of the iceberg on this, and that we really need to be diligent. And so there've been a lot of conversations here at this conference because the timing was just convenient. And everybody's trying to understand, you know, multiple parts of this problem.

How can we help prop up, you know, these single maintainer projects Mm-Hmm. Um, what, what models can we do, you know, alpha Omega project that the open SSF has been trying to solve elements of this for a while, right? To help fund, uh, people to do, uh, improvements on popular projects.

So that as something we've been doing, but we need to maybe do it at a larger scale. Um, you know, the other aspect that's come to light is the coordination between these things is very difficult. Hmm.

Um, you know, when you're, when you're talking about foundations like Apache Open SSF Eclipse, no, interesting. There, there are, um, there are groups in there that can talk about these things, but when you're a single maintainer project, by definition, you're not at a foundation. So who do they reach out to?

How do they get help? How do we reach out to them and share this intel that we now see this pattern? How do we reach them?

These are all problems that we're just now starting to grapple with. In, in addition, how can you try to vet more maintainers? How c can we provide capabilities to these projects so that, you know, if somebody shows up on the scene, can they reach out to us to, to give them assistance to, to check their backgrounds by whatever means possible.

You know, these are all very sober conversations that we're having a lot Of variations to this. There Is, it's a, it's a very complicated, Is it true to think of it as projects, single maintainer projects have been around for quite a while, where you've got some maintainer who's really, you know, been, uh, putting a lot of blood, sweat, and tears into this and needs the help? Or is it a threat also to more recent, even newer, And I think that the threat generally exists everywhere.

Right. And but, but clearly the easiest place for them to, to, to focus on are these single maintainer projects that Were vulnerable. Right.

One might argue that because the open JS one was detected and not allowed, that the foundation model helped in that case, because the attackers had to convince 10 or 20 people. Not all of them were maintainers on the project, but they were governing part of the foundation that were like, wait a minute, this is fishy. Versus a, a single maintainer who was just like exasperated and, and loving the fact that anybody is showing an interest.

Yeah. And, and that's part of the problem, right? So, um, so we, we have to figure out how to solve that because the, the, the challenge is these single maintainer projects, it doesn't mean that they're not popular.

In fact, that's where the risk gets consolidated. And the, and the, and it's a, an even deeper problem because many of these projects have been around for a long time, which means they're more or less done by some definition. Mm-Hmm.

There's not a lot of, uh, interesting new sexy feature development that developers wanna work on when these projects get old and they're popular and everybody depends upon them, stability, updating for new security things, updating when new standards are. Like, even think about like digital signing standards change. Mm.

You know, there's a lot of scut work that has to happen. Credibility testing and regulat All offensive. Exactly.

So it makes it not as interesting. Uh, it's not interesting work to do, but yet it's important work to be done. And so it sort of creates this problem where nobody wants to, nobody's excited by that work.

Um, and so it's hard to build a community. You could pay people to do that in theory, but that becomes hard too, because it's maybe not a career builder. You know, if you bring some new engineers in and you teach 'em to do this, like a, the work is not e it might need to be done once a year.

So it means in order for that to be a full-time job, you gotta work on a whole bunch of disparate projects. Right. Which means it's harder for you to become an expert in those projects, but also it's just a job.

And so if you get a better job, all of that knowledge walks out the door. And so then you'd have to backfill 'em. So there's a lot of problems with this that it's almost working On technical debt in a way.

Right. It's very much technical Debt. It's, it's hard to say, here's a big shiny thing to put on my resume That I did for them.

That's right. And everybody that runs engineering organizations knows that a model that just has developers only fixing bugs forever is not a great model. Mm-Hmm.

Because nobody likes to do that. Yeah. You can't keep those folks.

And, and I think we're gonna have to grapple with that type of problem in open source, because again, while it's hard, our society depends upon these things. You have to find a solution to it, even though it's a hard one. And it's not gonna be a single solution, because people tend to think open source is, you know, one, one problem to solve.

It's 10,000 things because you're, at the end of the day, you're dealing with individuals and not everybody wants To help their lives, their passions, their Commit, their passions, time. They may not want the help. They may not want to do things, uh, other than anonymously.

Right. So there's gonna have to be variations of, of solutions to this problem. And ultimately, transparency could help, you know, if somebody wants to create a project and only work on it anonymously, that's fine.

But, but people that are using the software should be able to know that and choose. Am I okay with, with that? Maybe I am, maybe I'm not.

Right. That's where transparency around and visibility around this. Do you think doing an analysis to say the more widespread something is getting used, 'cause it might be around for a long time or just very popular, is a much greater threat, uh, for a single maintainer project, or No, it's, we're just as vulnerable on something that may be kind of catching fire and popular early, early in its gestation period, if you will, of, of evolving and innovation.

Yeah. I think popularity and use is a, is a great surrogate. It's the place we should start.

That's why everybody talks about critical infras. That's the low infrastructure fruit to go at. And critical.

Yeah. It's the, it's the low hanging fruit that has the biggest risk and Yeah. Um, and even then, there's still a lot, but you know, you might be talking a thousand, couple thousand projects versus millions.

Right? So it's a, it's a, it's a tractable size problem that, that is solvable. A thousand, a couple thousand is still a lot of people, but you're not trying to solve every single project everywhere, because many of them don't, don't matter.

And that is going Mm-Hmm. So that's certainly a way to tackle his problem, but, uh, or at least it's a way to prioritize it. It's not necessarily a solution.

It just points the way to where you would start. Right. So there, there's helping people out who need, need the help.

Um, how about what can we do to equip people to recognize more easily when something like this might be beginning to happen and Yeah. They can be more suspicious, you know, kind of the, just awareness of okay, Right. I need to call someone and ask.

That's right. And that, that's what the point of that press release was in an attempt to get the information out there. I think we need to reach out and make it better understood for all these maintainers.

They, they may not be paying attention. They're by definition not part of foundation. So reaching them is gonna be hard.

But we want them to be aware that this is happening. That you might be a target and it might not be happening just now. It might have happened a few years ago, so you might need to look back in time to see did the conditions of, of something, uh, three, four years ago, did they match this?

If so, you know, you need to raise your hand. Who do you reach out to? Well, that's part of the thing.

And we're trying to create some places for, for these maintainers to reach out to through the open SSF, um, to help. That's an area that we can help. Um, they're not alone.

So getting that message out to those maintainers is really important. Also, getting the message out to consumers that this is what's happening. This is the new reality.

We are dealing with intentionally malicious components. You know, this is something that we've been talking about. Mm-Hmm.

For a long time. You know, Sonatype's been trying to solve this problem, um, for our customers Going, we knew it was possible. Right.

It is possible to do. It's being done. That's right.

It's possible to do. And, you know, we still bump up against organizations that don't recognize that there is this new threat. They're still trying to solve, you know, the log for J problem, right?

Mm-Hmm. Where I have a vulnerable component, um, but I don't know it, and I have to do an update. That's a very different problem than something that is designed to cause harm as soon as your developers download it.

Right? Right. And so that's what we've been doing.

We've been trying to provide solutions for that. You know, in our, in our report last year, uh, I think the number was, which Is a great report by the way, we'll have your, Yeah. com/s SCR.

Um, you can read it, but, you know, we've tracked 350,000 malicious components. In fact, we just published a blog, uh, this week. There was, uh, 15,000 in the last week that were flooded onto M Mm.

Uh, so, so, and, and that's a different part of the problem than what we're talking about these, uh, sleeper cells, as you call it, uhhuh. Right? So this is a dramatic escalation, but the solution to these is somewhat the same, making better choices around the projects, having the visibility.

So, you know, I'm depending on a bunch of projects that part in part of foundations that maybe don't have active maintainer, uh, status, you know, um, that's, that's part of the solution. Unfortunately, many organizations don't even know what they have, so they can't even start to look at the information about it. Right.

So we have to solve this problem by understanding what you have, then start making intentional choices about it. Then at least when something like this gets uncovered, you can immediately assess if you're affected. Mm-Hmm.

Right. And in, in instances where it's known to be malicious upfront and you're just starting to use it, you know, we can help block it so it doesn't even come into your organization. So there's multiple steps to this.

Um, you know, sunlight is the best disinfectant, I guess. Very true. That, that's why I'm out there speaking about it.

'cause I want people to know about this. Well, There's also the remediation part of it. You know, it may be as, I don't say simple, that's, that's a inaccurate way to, to describe it.

Maybe as basic as taking some things out, getting back to a secure code base or whatever. It could be much more incestuous. You don't know how deep it goes.

That's Right. Are you better off rewriting it? Or maybe there's a whole different strategy to employ.

It's, it, it's going to depend, you know. Yeah. An instance of xz.

Fortunately there was, these were the latest two versions. They were caught before. They were, uh, widely disseminated for many Linux.

It was a beta, i I believe, of a, of a Linux distro that this was uncovered in. So it was fairly easy for everybody to step back to the previous versions because these were like the newest, nobody really baked them in for a long time. If it turns out in the future we find that there's one that's been out there for a long time.

Yeah. That's gonna be much harder. Mm-Hmm.

It's going to look a lot more like open SSL the heart bleed problem, which was what, 10 years ago now? Can you believe that? Yeah.

It's, it is amazing. Yeah. Right.

Um, you know, where, where that was baked in that was not malicious, but it was a vulnerability baked into millions of devices everywhere. And still, you know, I think there, there are, there are sites that monitor this online. There's still something like 30 40% of the instances People made it at the firmware, I think in some, in some cases.

Exactly. And They never get updated. Right.

So if and when this happens and we find that these are intentionally malicious, it's going to be a massive scramble. And this is why having SBOs, having that understanding of the dependencies is the only way to prepare for it. You might not be able to prevent it because we can't know everything, but you can know what you have.

So when it comes out, you can remediate it as quickly as possible. 'cause that's where the real risk is. Right.

The longer it's hanging out there. Exactly. The longer Exactly.

The attackers have to export, It's been spread, et cetera. That's right. How, how Much attention is this getting visibility to the ciso, uh, side of the organization?

Not every CISO has, has been around open source projects enough to really kind of realize well, maybe what that some are. Yeah. But if you can increase something, but a lot of folks are not, you know, open source followers their whole career.

Maybe involved in, I mean, if, if you ask me not enough Mm-Hmm. Right. Um, you know, like I said, many of them are focused on trying to, to solve the log for J problem, which is great.

If you don't know what's in your software, you gotta start there. Yeah. But I think there, there is a belief in some, some pockets, many, maybe many pockets that if I can solve that problem, it will defend me against these other things.

And what I'm trying to raise awareness is it's not Mm-Hmm. Right. When you have these malicious components that will drop a back door, exfiltrate data smash and grab style attack, as soon as your developer downloads the component, It's Over checking it in your supply chain after, before you cut a release.

It's too late. Yeah. Right.

So the analogy I like to use is, you know, when you talk about demming principles and things like that, those are designed to produce better cars, better, safer, cheaper cars, and you should do all of those things. But doing that doesn't necessarily solve the problem If somebody's intent on blowing up the car factory. Mm-hmm.

Right. The way you would defend against blowing up a car factory is different than how you would make a better, cheaper, safer car. And that's what we're talking about here.

Interesting. And so we need CISOs to recognize that they need two parts to this problem. You need to do that because it will help you deal with the vulnerabilities that are gonna happen.

But you need to be focused on this malicious attack side effects. It's gotta be difficult in a CISO role to know how big of an issue is this for men and how do I work with the software organizations depending on the level that relationship, you know, kind of stay outta my stuff here, or That's right. What do we do?

Let's work together on this. I didn't realize this is that big of an issue. Yeah.

And you know, the ones that are most effective are the ones that actually work together with the dev teams. Because I think, you know, given all of the stuff we just talked about, if I, if I explain to you that it's a justifiable tax to spend more money to be safer in your security, I think everything we just talked about would, would justify Mm-Hmm. That that's a reasonable decision to be made.

Mm-Hmm. But the studies that we've done, and we're not the only ones, you know, again, part of this, the software supply chain report, you know, we found that organizations that work together to solve this problem with development, that, that say we're gonna be, you know, we're, we're gonna take account of our entire supply chain and do this. Right.

They are actually more secure than the people who say, we only care about being secure. Even if it means it's go slow and they actually can release faster than the companies who say, we don't care about security. We want to go fast.

So the trade off doesn't actually exist. We found that you can actually solve both problems and it makes you better in both dimensions. And it, it sounds counterintuitive, but if you think about it for a minute, if you only care about being secure, it means your process and everything is not designed to move fast.

Right. So when something happens, you're not prepared to respond quickly. Conversely, if you say, we are only focused on shipping stuff and we're gonna go super fast, you don't get to take a pass when lob for Shell happens.

You still have to stop and fix it. But what it means is you don't have the controls in place to actually deal with it. Yeah.

So on average, that stuff is gonna slow you down. When you solve for both of these problems, you're set up to detect, you're set up to respond, and you're set up to be agile. Right.

And so that's the message that I think a lot of people need to hear, that you can solve this, it's not a tax, it will actually make you, in some cases 30% more efficient, more innovative, however you choose to spend that extra time and energy ends up being better business value. It's not a tax, It's essentially something that I've talked a lot about with folks here at the conference is in the security world, we've learned it's not all about defense. It's equally, maybe even more so about response, right.

Because it's Right. It's not if it, it has and will continue to happen, right? That's right.

We gotta be able to handle it. In some ways we've kind of been doing DevSecOps the same way, or, or software security. And same way, let's defend, let's create more secure software.

Let's protect the supply chain. Let's do That's right. That's all good.

Um, but one of the benefits of being able to release quickly, respond quickly in, in a way repeatable, reliable, quality way, secure way is not necessarily, so you can deploy in production five times an hour. Not everybody is or want or could even do that. It, you may be releasing once a week or once a month, but the day you find out, you need to be able to take action and that's right.

In minutes or hours. That's right. If you found out that there was an intentional, malicious component in the SSH Damon of your Linux systems and it took you three months to update that, that's a problem.

Yeah. Not, not Every throw has to be a, you know, a hundred and whatever mile an hour fastball, but you gotta be able to throw one Meter. That's That's right.

That's exactly right. Yeah. It's interesting.

Well, I'd love to continue this conversation with you as this evolves. And unfortunately the sense is it, it will, it has been more to the story. It will.

Yes. Yes. Yeah.

Well, thank you for running this down and, you know, the, the threats continue to evolve and what's interesting is it's been there for so for a while, quite a while. How long you really Yeah. Well thanks for having me.

Happy to talk. Yeah, happy to, happy to have you. And we'll, we'll chat some more about it.

Great. Okay. Brian Fox.

Um, you know, it's, it's pretty, it is very interesting stuff and it's one of those that I think all of us staying up to date and collaborating with each other. 'cause no one person knows the answer to this of how big the problem is, more or less what to do about it. And I think the security teams are necessarily gonna know in all cases how to work with open source teams.

So that's where experts like Brian and others bring that to the table too. So, thanks sir. Thank you.

Appreciate it very much. We have more great, uh, guest conversations to happen here at Open Source Summit. Mitch Ashley, come back soon.

We see and will admit.