Open Source Security Challenges with Donald Fischer and Brian Fox

This is Textron tv. Hey guys, thanks for the tour. We're here with Brian Fox, who's the CTO for Sonatype, and Donald Fisher, who's CEO for Tide Lift.

And each of them has been doing a little bit of work in the space of, uh, study and what it is that open source maintainers are challenged by when it comes to securing their software supply chains and all the vulnerabilities they gotta deal with. Brian Donald, welcome to show. Thanks for having us.

Alright, let's start with Brian A. Little bit. I think maybe this whole conversation needs to start with a little empathy.

Not all of these open source maintainers are paid for their efforts, and a lot of it is a labor of love, and they're not exactly on call every time there's an issue. And so how should organizations that are dependent upon open source software kind of think about or approach these folks when it comes to security? Yeah, that, that, that's a very meaty topic.

Um, so, um, it, if, if we back it up a little bit, I think yes, empathy is, is an important part, part of this that I'd like to emphasize. You know, in this year's state of the software supply chain report, it was the 10 year tent annual. And so we did sort of a look back and one of the findings we, we found that relates to this topic was how long it takes open source projects to remediate vulnerabilities in their dependencies.

And 10 years ago it was, uh, something, I think it was about 17 days. Now it's 300 days on average, which is, which is pretty terrible. Um, over that time they've been releasing faster, which is great, but fixing these things slower on average, which is not great.

But, and this is where I always stop and point out, this is not an indictment of those maintainers. This is more of a cry for help because what's going on at the same time, if you look at the number of the CVEs that have been reported in the world and against open source at large has also skyrocketed right at, um, massive growth. Um, and the number of vulnerabilities, um, you know, I'm looking at the chart now in 2023, it was like 30,000, whereas 10 years ago it was in the thousands per year.

Right. So, and, and I think Donald, your survey found that the maintainers during this time are spending, you know, significantly more time fixing the vulnerabilities and, and dealing with this. So it's not that they're not trying to, they're trying really hard and still, you know, being pushed underwater.

And I think that's where some of the findings from the Tideless report other findings from our report kind of, kind of point in the right direction. You know, specifically we, we looked at the projects that Tide Lift has supported. Um, we looked at projects that are foundation supported, and we found that the paid maintainers, you know, are, are doing a significantly better job than the average, um, uh, uh, of addressing those, uh, those vulnerabilities.

Right. So, Donald, do you have more specifics on that? Absolutely, yeah.

And, uh, it's interesting because, you know, uh, Brian referenced the, uh, sonotype, um, state of the open source supply chain report there, which I kind of think of as a demand side view of this, uh, uh, problem space. The, uh, tide lifts, uh, uh, survey that we recently published is really from the supply side. It's focused on open source maintainers, uh, we call it the state of the open source maintainer report.

And we did find that, um, maintainers are spending a lot more time on security work specifically than they did just a couple of years ago. So in fact, we found that maintainers report that they're spending almost three times as much, uh, uh, time on security work than they were in our previous, uh, uh, versions of the, uh, of this survey that we did in the last couple of years. Um, now, you know, Brian, uh, referenced that their report found that commercially supported, uh, uh, uh, open source projects have, uh, you know, better, uh, security hygiene, faster time to remediation.

We also see that from, uh, the supply side when we look at the supply side. So specifically we found when we ask open source maintainers, what specific security practices do you assure for your open source projects? We found that aid maintainers, uh, undertake a significantly higher, um, number of those secure development practices than unpaid maintainers across the board.

Paid maintainers are 55% more likely to report that they're following critical security and maintenance practices, such as having multi-factor authentication enabled on all of the tools and systems they use to develop and publish their software. Um, using code analysis tools to look for potential security vulnerabilities, um, having a, uh, security response policy published, and a clear allocation of responsibilities for what happens if a security vulnerability comes up. So, um, overall maintainers are spending more energy on, uh, the security of our software supply chain, which is good.

And the really good news, I think, is that paid maintainers are doing it, um, even more to a higher, um, standard. And that's great because that means we can pay more maintainers, more money to do more of this. We have a way to influence the health and security of the open source comments that we depend on.

And just outta curiosity, Donald, how many maintainers are paid or want to get paid and what's the challenge with paying? Yeah, absolutely. Um, so, you know, this is one of the, the things that we, um, uh, highlight in, in the tide lift, uh, say the open source maintainer report.

And it's a really key finding that is counterintuitive, I think, to a lot of people in the, uh, in the industry, which is that, uh, of the hundreds, you know, we surveyed over 400 independent open source maintainers, uh, for this, uh, survey. And over 60% of them reported that they are unpaid. They're, uh, they do not receive any kind of compensation associated with their open source work.

Um, 25% of the maintainers reported that they're sort of a semi-professional maintainer. They're earning some income somewhere, um, as so associated with their projects and only 12% reported that they're a professional maintainer. Most of their income comes from their open source work.

So, um, a lot of folks just don't have that intuition, especially organizations that are consuming open source, they don't really realize and they kind of assume, well, maybe Google's paying for it or something, or there's a company around all these things. Uh, the majority of the widely used open source projects are, uh, uh, looked after by unpaid maintainers. But there's a really cool opportunity here, which is the vast majority of those unpaid maintainers told us that they are interested in getting paid to, uh, spend more time doing more things around their open source projects.

In fact, um, you know, of the, um, if you look at that 60% who report that they're unpaid 44% of the total, uh, maintainer, uh, audience, uh, report that they're currently unpaid, but would appreciate being paid, uh, for undertaking, for example, secure development practices for their projects. So again, opportunity, we can do this, all we have to do is, uh, you know, kind of act. Yeah.

Ryan, what is the challenge in actually getting the tools to these folks and maybe the training? I mean, we've been talking about this issue of, uh, making software developers in general more aware of security issues, but I feel like, um, you know, sometimes it sticks, sometimes it doesn't. And what is the challenge here with getting these maintainers who are really developers in their own right, just more security conscious.

Is that all there is to it? Or is there something more that needs to be done? I mean, we, in the software industry for the last 50 years, we've been trying to make engineers better at security, full stop, open source commercial, irrelevant.

And while I think that's helped, certainly, um, it's clearly not solving the problem and, um, you know, just more of the same, we should expect more of the same lack of results, right? And so I think there, there is a little bit of, um, uh, a lack of motivation on the consuming side of this that is driving some of these behaviors. So, um, and this, this is why I've been a proponent of, you know, the software liability changes.

We're finally seeing some of that in Europe. But I think, I think there's a sort of a, a, you know, a trickle down effect or a, a, a cyclical kind of kind of relationship here, right? So if, if slash when organizations are finally really held accountable for the choices that their developers are making in terms of the, the projects, the open source projects in this context that they're using inside of the software that their company is selling, until that really becomes something that is significantly, um, you know, of risk for the, for the business in terms of a, a major lawsuit, we're gonna see the same thing.

And so when those organizations make those better choices, then I think you will start to see open source projects actually caring about that. Right? Now, I can create an open source thing.

If it provides a little bit of value, I throw it out there, you know, millions of people might use it in some future state where that won't be successful unless I follow some of the steps that Donald's, you know, projects lay out that I can attest that I'm doing things securely and that, you know, I'm, I'm following all the practices, my project won't get traction. So in other words, the natural evolution of that will come only when the consuming organizations are basically using that to select better projects. There will be a natural selection, in other words, of those behaviors.

I just don't see anything that's gonna change the game. It's not a lack of knowledge. Most open source maintainers these days understand that they should be doing it.

They're not getting the support. Sometimes the tools and the infrastructure required to actually do these practices, they're not free. Somebody's paying for them somewhere, right?

And so, um, so, so I think that's part of the problem here, that we can't just tell everybody do better, frankly, we've been doing that for 50 years and it's not better. Yeah. And, and also by the way, you know, when, when those organizations are on the hook to actually think about that, suddenly, you know, paying the maintainers a little bit of money may sounds like a tremendous amount of value.

You know, I'm looking here at the statistic. This comes from Harvard Business School. They estimated that the open source produces, you know, about four between $4 billion on the supply side and $8 trillion of economic value on the, on the, the sales side of this.

So in other words, the open source infrastructure that is being created and shared leads to an $8 trillion economic outcome. Why can't we peel off like a fraction of a percent of that to help support those very people? When that happens, these things will get solved.

Until then it's, I think it's an uphill battle. Yeah. And we have some, uh, you know, from the tide lift, the, looking at it from the supply side, from the maintainers perspective, I already shared that, uh, maintainers report that they're spending three times as much time now than in years past on security related tasks.

So some of that message is coming through, starting to come through from the demand side, the user, the organizations that are using their software. We also pulled maintainers specifically on, um, their familiarity with specific secure development standards or regimes. And we did find a notable increase in awareness among open source maintainers of standards such as the US government, uh, uh, promoted NIST secure Software Development framework.

Um, last year when we did our maintainer survey, only 26% of maintainers were familiar with the NIST SSDF, whereas, uh, this year, 20 24, 30 9%, 26 to 39, so big swing in awareness there. We also asked about an industry, um, initiative around, uh, open source software security practices, uh, coming from the open SSF uh, uh, the, uh, you know, group of, uh, uh, industry participants, uh, around open source security. And from last year to this year, we saw an increase in awareness from 28% to 48% among maintainers in the open SSF scorecard and their sort of rubric for secure development practices.

And the number of maintainers who reported they had heard of none of these secure development, uh, standards went down from 52% last year to 40% this year. So I think that's a sign of green shoots. You know, there is some, not only opportunity, but we're starting to make some progress connecting the needs of the organizations that build with open source to the creators of open source, even if they're not traditional companies or even foundations.

Um, and just a point on, on that last one, you know, one of the things that we found is that the, the, you know, we reverified that the vast majority of open source maintainers report that they are solo maintainers. They don't work for a company with a bunch of colleagues. They don't even have a bunch of open source maintainers.

The majority of, uh, open source maintainers that we spoke to are single maintainers of their projects. They're, um, you know, uh, and of one looking after each of these essential projects. Um, so, um, yeah, So Brian, about once a year or so, there's this, you know, moment of panic when some zero day vulnerability thing gets discovered and everybody goes running for cover.

Um, do we need some sort of standby SWAT team that's available to help these maintainers when those issues arise? And or is there some other way to think about this? 'cause it seems like that army of one is quickly overwhelmed whenever there's a crisis.

No, I don't, I don't think so. Um, I think I, I strongly believe and don't have the data immediately at my fingertips to prove it, but that most open source responds quicker than commercial open source. I mean, take, uh, sorry, than commercial, uh, uh, code companies in terms of turning these things around.

I mean, log for Shell is a, is a really perfect example that highlights this, right? That team turned around the vulnerability when it was irresponsibly disclosed in Asia over a holiday weekend, right? They fixed it in days.

But we've, I know you and I have talked about the uptake of the fixed versions of Log for J over the last four years. Um, this year we reported it's finally we're down to 13%. So 13% of the time, um, organizations are still downloading the vulnerable versions of Log four J As of a couple months ago.

Last year it was 30%, right? And so we're talking about the most, um, high profile, most discussed. The, the vulnerability was on the, um, the top 10 nation state attack vector list for the last 10 years.

And consuming organizations are still pulling it down in a non-trivial amount. And they, and it took them three years, like I said, last year, it was still one third. And so the fact that the project team turned that around and fixed it within days is awesome.

And yet the impact lingers, right? The long tail of the updates. And so we looked at, um, enterprise software, we found that on average, 80% of dependencies are unup updated year over year.

We've also found that for the last three years, this statistic has not changed. Um, 96% of the time when something is being downloaded and included in a build, um, that has a known vulnerability, it's already been fixed. So only 4% of the time are consuming organizations, including components where they have no choice because the open source hasn't fixed it, right?

So all these things have to happen simultaneously. We need to be able to, of course, support open source, help them, uh, respond to these things. It's only gonna get worse when the legislation comes, 'cause the demands on open source are gonna get higher.

So they definitely need help there. But the real problem at the moment is still consuming organizations by and large are doing a terrible job of managing their supply chain full stop, right? That's why we started writing this report 10 years ago.

And while many dimensions are improving, it's clearly not improving fast enough. It's not improving as fast as the attackers are. Um, certainly.

And, um, I think that is the reason why we're starting to see regulators worldwide step in because the organizations, the industry has not appropriately responded. So I think that's where the problem largely lies, Right? Donald, to Brian's point about that, the maintainers will say, I did all the right things.

It's all these other folks that keep downloading the wrong version of these things, and it's not my fault. So, um, how do we kinda bridge this gap? Yeah, I mean, I think there's a couple elements to the, um, to the solution and, uh, you know, uh, this is, uh, a large part of our focus at Tide Lift is how do we bridge the interests of these organizations that are consuming all of this, and those open source creators who typically didn't start their open source projects because they wanted to become part of the software supply chain of large organizations, but they've wound up in that, in that circumstance, right?

So I think one element is agreeing on what good is. And I think that that's a, there's a positive development with these, uh, standards that we talked about before. Government standards, like the NIST Secure Software development framework, industry standards like the Open SSF scorecard.

They're literal checklists of, okay, here are the things that we want to have done for the open source software that we consume. It makes it very concrete, it's a clear ask to the open source maintainers. But I think we've found now we've verified as an industry that just asking is not enough, um, because there's a cost to doing it.

Brian referenced this, right? So, um, a, you know, simple capitalistic way that we can approach this problem is if we're making an ask that's not free of cost, um, but, you know, not tremendously difficult to get done, let us bring an economic exchange into the, into the picture and say, here's the list of things that we want you to want, we want you to do. And, uh, you know, back to you, uh, something you said, Mike, first of all, they, there's no appreciation here, so thank you for doing what you've already done.

And we want you to go above and beyond that. So how about we can, you know, pay you this much money? Maybe it's not a new full-time job, but it's some, you know, non-trivial income on the side in exchange for going and undertaking this list of secure development practices or, you know, other improvements to your software that you otherwise wouldn't do for free.

And we've seen tremendous success with that model over the last couple of years at Tide Lift, starting with just saying thank you and then making a clear ask, providing some financial incentive. I think of it as like a nudge, you know, just to kind of like give you a little excuse to go the extra mile on this stuff. But the impact to organizations that consume this software is tremendous because they can get a clean bill of health across that entire, you know, uh, standardized checklist.

And that's really what they need to understand what, what they're building their software applications out of. Brian, is it always cash that people want or are there other ways of driving compensation? 'cause what I think cash, they're not gotta fill out tax forms and all this other stuff.

So are there other ways to think about compensating folks? Uh, I, I don't know that I have the answer to that question. Cash certainly complicates it.

Um, and, uh, Donald probably has some thoughts here, but, you know, certainly help is going to be required, right? Help can come in many forms in a single maintainer project. Cash to, to turn, you know, that pet project that is a nuisance into an actual side hustle is clearly demonstrably based on our data useful.

I think in, in other cases, especially as we start to see more stringent requirements coming, you know, the Cyber Resiliency Act and, and things like that in Europe, um, we're going to see first, you know, potentially tone deaf requests from big organizations, emailing the open source maintainers as they would a vendor demanding they fill out a survey and that's gonna get them nowhere. But showing up and actually helping to create the documentation, helping or providing resources, you know, people, employees to help, uh, with some of these things, or maybe even in infrastructure or donating tool licenses to help. All of those things are, are required and they all happen a little bit.

Um, but there's, there's many ways that organizations can, let's just say, invest in the open source that they're dependent upon, um, and they can take many of those different forms. Donald, last thoughts on that. We, we have data on this again from our survey.

So we just straight up asked open source maintainers, you know, we know there's a lot of good aspects of being an open source maintainer, but what do you dislike about being an open source maintainer? And the top three things that they reported, number one, is not being financially compensated enough for their work. Uh, number two is feeling underappreciated or like the work is thankless.

And number three is, it adds to my personal stress. So I, I do think that they are telling us the answer here, which is that, yeah, money actually is, uh, a significant, uh, portion of this, but it's not all money. I mean, you know, number two is they feel underappreciated.

I think it's also appropriate for us to find ways to say thank you and demonstrate our, um, gratitude to the open source creators. And then around the third one adds to my personal stress. I think that's where you get some of this.

Um, yeah, how can we help share the load? How can we, um, provide tools, as Brian was talking about, how can we, um, offer, you know, uh, for those solo maintainer projects, how can we contribute organizational support of, uh, a new co maintainer or help, uh, you know, resolve some of those requests. But the number one, uh, solution that maintainers are proposing when we ask 'em this question or the number one pain they're looking to address, is that a lot of folks are making a ton of money using their open source projects and assuming that they're gonna do all of this work to bring it to the enterprise grade and they're not getting TED in on any of it.

And that's a really straightforward one for us to be able to solve now. All right, folks, you're hearing it here. One, open source maintainers, no matter what you may believe are not standing by waiting for your phone call and two, compensation still drives behavior.

Hey, Brian, Donald, thanks for being on the show. Appreciate your time. Thank you.

All right, and back to you guys in the state.