The Open Source Trap: Why Trust Isn’t a Security Strategy

Hey everybody, we're here in Amsterdam at the KubeCon plus CloudNativeCon Europe Conference, and I'm talking to my friend Josh Bressers, who's from Anchore. Yeah. Hey, Mike.

Hey, welcome to the show. Thanks, man. I'm excited to be here.

" It was a topic that we kind of used to nod to, but now, everybody's seeing these npm packages. There's all kinds of attacks coming through, and suddenly it's all very real. So what are you hearing from folks about what they're doing about all this?

I love the question because we're literally in the middle of a supply chain attack right now against an open source scanner. It's not one of mine, thank goodness. But this is a really hard problem because we don't know what to do in a lot of instances.

So I've been doing this open source thing for 20 plus years. I joke with people that I was doing open source supply chain security before we knew it was called a supply chain. So it's been a long time.

But I think how we handle open source security is, I wish I had an answer that was just run this one scanner, buy this one product, do this one thing. But it's not one thing, right? Because when we talk about npm, the problems we see in that universe are very different from the problems we're seeing when we talk about Go or Rust or Python.

And there's a certain amount of cooperation we get between these groups. Like for example, we're here at KubeCon, you've got Linux Foundation has an OpenSSF, and the OpenSSF has groups, for example, that brings some of these people together where you have the package repositories starting to cooperate and communicate on malware in their infrastructure, on vulnerabilities in the packages and how we find them and how we detect them. And so quite frankly at this point, if this is something you're concerned about, this is a lame answer, but we kind of have to go back to the fundamentals of security, is just understand what you have, scan your existing infrastructure and software for just the easy vulnerabilities.

Not even the crazy stuff, just the easy stuff. Because we're bad at that too in many instances. And so start with the fundamentals and then we can kind of work our way up and start having discussions about how are we going to secure the packages we're using from open source?

How do we keep track of the packages we're using from open source? Because one of my favorite examples is if I asked you what is an end of life open source package, what would you tell me? Maybe one that hasn't had an update from a maintainer in five years.

I don't know. Five's a pretty good number. A lot of people say one or two, but there are many legitimate projects that haven't had updates in five years, and they may not have an update for another decade, not because there's anything wrong, just because they say, "It's good enough.

" Right? And so it's hard to figure this out. There are projects that will mark themselves archived or end of life, but we can't really define this.

" The episode came out yesterday on Monday where I talked to Brian Fox from Sonatype about their 2026 supply chain security report, and they do this every year. I talk to Brian every year. I've known Brian for a million years and I love him to death.

And we talked about end of life and the fact that just defining that, it feels like it should be really easy to do, and it's almost undefinable in open source, which is, again, one of the super frustrating problems we have when we try to define the whole open source security universe and what's going on. Mm-hmm. I know, right?

That was a mouthful but like- No worries. Is it your sense that the adversaries are just getting smarter about how to exploit these issues? Or have these issues just been around forever and they've now found the time and the tools to go and actually do something about them than when they were too busy doing other things before?

So I think there's a couple of pieces at hand. None of this is new. These are the same problems we've always had as long as I've been doing this thing.

But I think it's not that the adversaries got smarter. I think they started paying attention because if you look historically, a lot of the adversaries attacking organizations, backdooring components, taking over infrastructure, there was a heavy focus on a lot of the commercial software, but it's the popular commercial software, right? It's kind of like the old saying, why'd you rob banks?

Because it's where the money is. Opportunistic adversaries are going to go where they can get the most bang for their buck, so to speak. And so historically, the easy answer was often large commercial entities that had, we'll say, horrible security.

But now open source is in everything. The numbers vary anywhere between 70% to 90% of all software is open source, and it doesn't matter which number that is because it's a ridiculous amount, right? And everything is full of open source.

It's all over the place. And so this is a very opportunistic place that these attackers can start looking at to understand, what can I do here? And in some instances, it's pretty low-hanging fruit.

And I'm not faulting open source in any way for this, right? This is one of the challenges the open source maintainers have now is they're being essentially attacked from all angles where you have the bad guys attacking them. You have community people complaining about what's wrong, or there are bugs, or they're not fixing security fast enough.

You've got enterprises using their software demanding features. So it's coming from all over the place, right? But anyway, back to your question.

So I think when we look at most attacks, they're opportunistic, and this is where the money is, so to speak, right now. But also because it's open source, there often aren't resources, right? One of my favorite examples is if we look at any of the organizations over on that floor right there that have the big booths, they have security teams with hundreds or thousands of people on them.

And if we look at the data of open source projects, the vast majority, I'm talking like 90% plus of open source, actually, it's more like 99% of open source, is one person. Mm-hmm. They don't have a security team.

They don't have any team, right? It's literally them doing the work. And so I think one of the challenges the industry needs to start figuring out, and this is obviously a good place to have a conversation like this, is what do we need to do to start supporting some of this open source, some of these projects, some of these foundations?

There's a lot of work to be done, and it is happening. I'm not going to say it isn't. And especially in Europe, we have the Sovereign Tech Agency that's been doing an amazing amount of work.

The OpenSSF just got, I think it was $12 million for Alpha Omega. Yeah. Alpha Omega's been doing amazing work.

I actually have an interview with Michael Wisner about that coming out next week. So I'm very excited about that, and that's really fun, too. Some people would say that relative to the mission, though, those efforts, the dollars involved are a drop in the bucket.

So what do we really need to do to secure all this stuff? Because there are tons and tons of these maintainers out there, not all of which are especially passionate about security. Maybe.

I don't know if I would say that. I struggle with this one, right? " And they'll be, "I don't care.

I hate security. " And that does happen, but I don't think it is common, right? It gets the news because it's interesting, and we laugh at them, and we point fingers and make a big deal about it.

So I mentioned I had a podcast. I talk to a lot of open source developers. It's one of my goals is to hunt them out and talk to them and understand what they're talking about, because a lot of people aren't, right?

One of my complaints is that when you talk to a lot of organizations and even some of the foundations, I'll pick on them a little bit, is they'll tell you what open source thinks. And then you go talk to a lot of these developers, and that is not what they're talking about. Mm-hmm.

Which is an interesting chasm we have to cross sometimes, right? But now, your question about funding, is it a drop in the bucket? It absolutely is.

But that's okay, too, because we have to start somewhere, right? This is one of those situations, if we would just dump more money than we knew what to do with on top of open source, it would ruin it. It wouldn't fix it, right?

This is like where you hear the stories of people winning the lottery, and they're bankrupt a year later. So I'm not saying we should stop and say, "Oh, $12 million. Good job, everyone.

" But I do think it's a good place to start as we try to understand what can we do with this money, right? Because how we fund some of this stuff, what we fund, is a challenge. And I'll give you a really good example.

So in all my discussions with open source developers, there are open source developers that have, let's say, one or two small projects they work on, right? And they do it on the weekends, maybe one weekend a quarter at most, because they have lives and families and everything else going on. But when you have one or two small packages, basically no amount of money is going to get them more time, right?

Because they have jobs. They have families. If I say, "Here's $10,000," that doesn't help them, really.

Maybe it'll buy them a vacation, so now they're going to miss one of their quarterly updates. And the challenge being, these people need, if you want them to work on this, they need it to be their job, right? Where they're making enough money to support themselves, to support their families.

But now we go from, "Oh, I'll give you $10,000" to we're talking about hundreds of thousands of dollars, or I guess euros, being in the current venue. Yeah. But you know what I mean?

And so that's one of the challenges is there are developers we can easily say, "Here is a living wage. " But then there's other developers that might have one project. Who's going to pay them a living wage to work on their one project?

Do we need maybe a SWAT team that helps those one-person teams and that's their job to go help them handle this particular security update or fix this patch? Maybe. So that has been tried in the past more than once, and in fact, that's basically what my job at Red Hat was for a long time, is I would help coordinate security fixes amongst open source projects.

And there are projects you can do that with where they'll maybe accept the help, they want the help. But sometimes they won't take it because maybe the project they work on is kind of weird and obscure. But one of my favorite examples is I was reporting something to one of the font...

It was one of the TrueType font libraries one time. And I had all these reproducers that would crash the font library. And I put together a patch, and I wrote it all up, and I went to them, basically said, "Here's what we found.

This is what's going on. " Looked at my patch, and they were like, "This patch is terrible. " Because I thought it was three bugs.

It was actually one bug in some function I didn't even know existed. " Right? Because now they looked at my patch, decided I was an idiot, and then they went and did it right.

Do you think in the age of AI, it might be easier to create the patches going forward? Because now we have AI coding tools, so theoretically, they could tell us what we need to fix. Maybe.

I'm very interested in this topic. This is something I've been looking into. So I think the AI agents give us a couple of almost dichotomies when we look at them.

So they're pretty good at writing code, right? Right? Because if you look at someone who gives a bad prompt, you kind of get crap code out, but if you give it a really good prompt, you can get okay code.

Got you. Now, there are instances where the LLMs are writing patches, and they vary in quality greatly. I know there are people building autonomous agents that look through bugs in their GitHub repository and then creates patches for them and creates test cases for them, and it's almost magical in that regard.

How is the long-term quality on that? We don't really know yet. We don't have numbers.

So I'm really interested to pay attention to that. We also have LLMs that can find security vulnerabilities. Finding security vulnerabilities as a human is really hard to do.

Mm-hmm. I can probably count on one hand the people that I'd say are good at it. And so the bar's not very high.

And so I would say LLMs today are probably better than the average human at finding security vulnerabilities. And then we go back to now we have to patch them, and we have to see how the agents are going to do with the patches. And so that might be one of the answers.

I'm not willing to say yes or no yet because I let you know, prove it, I guess, is where I am. I like to call myself, I'm an AI skeptic, where I want to believe, but I'm also not going to make claims that aren't true because oftentimes that's worse than anything else. But are we also concerned that the adversaries will be using AI to create exploits faster than ever?

Because we've been lucky when you think about how long it takes them to do something. Well, maybe that's going to be now done in a few days, and they'll be launching all kinds of things that we're going to be- Yeah ... " Hours.

Not days, hours. So if you look, there are various graphs in the security world that show time to exploit. Right?

Where how many days does it take from the time a security vulnerability is announced until we have an exploit for that security vulnerability. And a couple of years ago, it was like 30, 40, 50 days. It was hundreds of days a decade ago.

Now it's literally hours. And without question, part of that is LLMs. They're really good at writing exploits.

" And it does. Well, usually. Not every time, right?

It depends. The answer's always it depends with this stuff, but it's pretty good. I'd say it's better than a human.

I've written exploits in the past, again, when I was doing my coordination, and it would take me days sometimes to actually get to the point where... Because you have to craft a JPEG image and then modify it in just the right way, and you test it and then you tweak it, and then you test it, and you tweak it, and it's just back and forth and back and forth. And these things can do that faster than a human can imagine.

So, and yes, adversaries are 100% using this technology. Adversaries, when we think about the defender versus attacker scenario, the defenders always trail attackers, just by definition. That's how it works.

I know we like to pretend we can do it some other way, but it's just how it is. A wise man once told me it's a lot easier to throw grenades than catch them. Yes.

Exactly. 100%. 100%.

Are the folks who write container applications maybe a little too comfortable? " And they have to go find that container again, and they may have the same code over again, but they'll never find it because it's just moving too fast. What's reality?

So I love when people ask me this because... So you've got one camp of people are saying you have to move as fast as possible, right? Update every 15 minutes.

Update it as absolutely fast as you can. " They're calling it a cool down period because, for example, a malicious actor might have inserted a backdoor, and let's give the community and researchers time to do their investigation and understand, is there malicious code in this update? And let's say after seven days, if no one finds anything, we'll assume it's okay.

Right? And these are very different problems because you're moving very quickly, and we're telling people not to update. But I think the reality is, if you look at the vast majority of organizations, and I talk to a lot of organizations, both big and small, most of them are not moving that fast.

There are some, but most aren't. And so this is where you get back to the fundamentals I mentioned way back at the beginning of this conversation, where understanding what you have, being able to run some scans against it, understand what are the potentially dangerous spots I have in my infrastructure, and then I can focus my very limited resources on those areas. " And that's just not most people.

Mm-hmm. The open source community is built on trust. It's at the foundation of the thing.

It was like everybody was pulling for everybody and doing the right thing. Are we in danger of losing that? Because now you've got bad actors who are joining projects and pretending to be people who are legitimate, and they'll work on a project for six months and then insert something nasty and do we have to kind of just assume now that not everybody is who they say they are, and their intents are not always benevolent?

That is one of the things that I talk to a lot of people about, and we don't have a very good answer. So there's two sides to this problem. What you're talking about is the XZ incident, where we had a rogue maintainer, Jia Tan, that-...

inserted a backdoor into a project, and it is terrifying. Because you know what keeps me up at night, Mike? Is that there is no way that's the only project that group went after, and we don't know who else they went after.

And that legitimately scares me, because I have a suspicion there is a backdoor somewhere we haven't found yet. We might someday, we might not. It's very hard to say.

Right? Now, on the other side, there's people who say, "If you don't want to trust open source, what else are you going to do? It's 70% of your application.

You're screwed. You have no choice. " And I think this is something we see where, one of my favorites is you'll talk to security vendors, and they'll tell you, "Oh, you can't trust public package repositories.

You need to buy our product where we're selling you trusted open source. " There's all these things. But the reality is, almost everyone is using all of the open source from all the public repositories, and there are people getting whacked every day because they're using untrusted or typosquatted packages or whatever.

But if you look at the numbers, it is almost unquestionable that the benefit we're getting from open source, even knowing some of it is malicious and some of it is dangerous, grossly outweighs the potential problems there, right? It's like we could sit at home in our hermetically sealed houses and never get sick, or we could come to a conference knowing that there is a possibility we're going to get sick. But that's okay, because the benefits we get from things like this vastly outweigh catching a cold or, hopefully, not worse.

" I wish I could say you could just do this. I wish that in a million years. Right?

Obviously, buy my product. That's the answer. But no, in all seriousness, I think the correct thing to do in this instance is the whole point of open source is that community aspect and understanding what's going on.

And I'm not saying you have to be involved in every project that you use, because many of us use hundreds or thousands, so that's not realistic. But I would say find something you care about and try to make it better. It could be one of the dependencies you use.

It could be joining a CNCF group. It could be working with OpenSSF. There are a ton of organizations.

There are a ton of projects. There's lots of projects looking for help. Go find one that needs help and get involved.

So I really think the trick to open source is being part of open source, not buying a product or setting a timeout on your cooldowns for upgrades, or running a scanner. It's really about get involved and understand. Be open source.

There you go. Folks, that's the secret to all things. Stay involved.

" Yes, 100%. Absolutely. Thanks for being on the show, Mike.

Thanks, Mike. All right, and we'll be back.