Episode 5: Cybersecurity – Safeguarding the Open Source Ecosystem – ActiveState Podcast

I am Nicole Schwartz. I'm a security product manager at Active State and this is Dana Crane. Hey, the topic for this one is cyber security, safeguarding the open source ecosystem.

That could mean a lot of things. I think what we're gonna be focusing on here is when you're consuming open source, you need to take into consideration the GI repo that it came from or other types of repo. The place in which you got it from that could be like NPM, Maven ipi, as well as bringing it into your system.

So kind of those three parts are I think where we're gonna be concentrating today. So We'll talk a little bit about what's happening with those ecosystems and how you can better safeguard against some of the things that might be plaguing those systems right now when it comes to things like supply chain attacks, So active state, why are we talking about this? We've been working with open source for over 20 years and we've got a lot of internal expertise that goes from how to build open source as well as how to patch open source and how to make it work within enterprise organizations.

I would like to say we're the open source management platform that can make you more productive and more secure. Now, why is it that open source is such a big thing right now, Dana? Like why are we even talking about this?

Well, I mean, open source has been adopted by everybody. You know, it uh, kicked off some 25 years ago, but it really got adopted when the US government said in like 20 15, 20 16 that they're all in. And since then we've seen a huge explosion in the open source to the point where almost everybody is using open source these days and it's composing 80% or even more of your code base.

So everybody's awkward with this and we see the huge explosion in usage. I think that's really the crux of uh, why it's become such a big target for hackers as well. And just from a usage standpoint, you wouldn't be using something unless it gave you some benefit.

And I think the benefit that we see is it gives you a lot of speed, cost effectiveness and also flexibility. You can transition from one community project to another community project to fulfill your function. So it really is very useful to everyone and I can understand why everyone's using it.

But you'd mentioned, uh, that it was under attack. I believe we have some pretty terrifying statistics we've seen for just how under a jacket is. Yeah, I mean it depends on what report you look at, but anything from 60% to 90% of all companies were apparently, uh, subject to a supply chain attack.

In the past year, we've seen stats from companies like Synopsis who have, you know, seen clarity than 33% growth year over year, I believe 2023, twice as many attacks as the previous three years combined. 1 billion in payouts and that's just the ones that are actually reported. They see up to 80% of ransomware payouts never ever get reported.

So it's a pretty scary landscape out there right now. It makes sense if you can attack one open source system or dependency that is used by many different people that from an ROI perspective of an attacker, it makes perfect sense. I believe Gardner said about two thirds of US businesses were impacted in 2023 by some kind of software supply chain attack.

So you're, this is more than 50% chance that you're gonna be impacted. What kind of attacks are there? You had mentioned ransomware, I think malware.

What other kind of things should we be looking out for? There's lots of things that are happening. The good thing is a lot of the ecosystems are starting to batten down the hatches.

A lot of them have a multi-factor authentication in place now in ways to make sure that you know, through API keys or through OAEC that you are verifying that your account is uh, well protected. But we still see a lot of things happening out there. I mean the traditional one that's been around forever of course is just typo squatting.

I stick an S on the end of name of a popular package and see how many people downloaded before it gets caught. Uh, people use malware scanners, but they're usually based on the existing known malware variants, right? If somebody's come up with something new or somebody's come up with a new variant, very hard to catch those.

So those are the kinds of things that we see in those malware. Could be, you know, it could be back doors, they could be Trojans, they could be just about any kind of variant you might think of. Hackers are nothing if not creative.

Speaking of backdoors, I think the latest exciting thing that people have probably read about is the XV utility backdoor. That one did not make it into the wild and really the ecosystem got darn lucky with that one being caught before it got merged in. And that one with the combination of social engineering where they put pressure on the currently overworked maintainer to hey, you know, there's a whole bunch of problems, are you gonna patch that?

Are you gonna fix that? As well as like offering to help and do patches. And then among all of those patches they snuck in the back door.

So if you think about when you do your security training at work, they tell you like the phishing link, how those kind of social engineer you into going and looking at that bad website and putting in your credentials and stuff. This is kind of the same thing. They're starting to use some social engineering mixed in with just your traditional scraping and malware.

Yeah, that's going right back to the community, right? Starting right from the very first point you get into the community and you start being a verified uh, contributor, but what they're contributing is actually bad stuff. Uh, so yeah, another thing to look at or there, I think the uh, interesting thing is as you mentioned, both your repositories as well as your package repose, your source repositories and your package repos are looking into following more common practices that people have at their day jobs of the two factor authentication and signing your commit.

I believe that pi pie has a request going in where they might start having SLSA attestations. As part of pi pie though, there's all of these kind of interesting things that are hopefully gonna help the ecosystem at least be a little more transparent if not follow best practices. But that really gets into, as we had talked about the start of the process, you've got people writing it, you've got a package repository and then you're consuming it.

What ways do we think people can optimize their consumption to do that in a more secure and trustworthy way? You've gotta be aware that most of the organizations out there don't really take supply chain management as a sort of first principle. It's kind of a thing that they tack onto their software development process.

So they're using multiple different tools. You've got a package manager, you've got environment manager, you've probably got an FCA tool, maybe you're dropping it all into a binary repository. All these things are sort of disparate tools that you've cobbled together to try to get you a picture of what you're importing into your environment.

But I think that's the key, right? You've just imported something, you've let the fox into the hen house and now you're trying to figure out where is that fox? How can I find them?

How can I make sure I know what it is so that I can get rid of them or patch it out or something. You're doing something in reaction through rather than being proactive about it. And I think that's the whole point, right?

You're not in the software supply chain business, you're in the software business, but your software today is more assembled than written. So you've gotta give some more attention to how you actually work with that software supply chain. And that may take uh, different approach.

And what we like to say is that active state can be your outsourcing vendor of choice when it comes to managing your software supply chain for you. Now, something you pointed out there is that a lot of this is reactive and if I can just recommend everybody listen to the podcast episode that I do with Martin, we go over kind of balancing development and security because you're right, there's not enough time and effort put in, there's also not enough preparation. A lot of times you throw people in and you say, okay, go find the fox.

Like good luck and don't give them necessarily a lot of help. We gotta cover the hotness. AI is the hotness right now and AI help me with this being more proactive than reactive.

Like where, what is the role here with finding these bugs or reacting to these bugs? Can it help me? Yeah, I've seen a lot of tools out there like well of FCA tools out there that have started adopting AI and one of the key points here is that most people are importing pre-built packages from the environment.

Some of them are actually going, you know, GitHub and pulling down the source. You gotta be a little bit careful when you go to GitHub. There's a lot of fork There could be type of squatting or the everything package, Right?

Right. Good points. Yeah, we've seen people do their malware fork bombs where they create another version of say, uh, popular package and they just listed under their own account.

You don't have to do typos clothing, in that case it's the actual name of the regular package. But you do have to use social media to convince people to download your version of it though it's a little bit of a trade off there. Uh, but yeah, so whether you're pulling in source code or whether you're going to a repository of pulling in binaries, which is more likely, you've gotta be able to scan that binary that you're pulling in those binary scammers.

They tend to throw quite a lot of alerts, uh, which has resulted over the past couple of years and along cybersecurity burnout because those alerts take a lot of investigation. So AI has come along and they've been trying to use AI to figure out what is a real alert and what is a fake one and sort of minimize those. There have been some success.

Hopefully it gets more successful because we really need to take it easier on our cybersecurity professionals. We're losing them left and right and there just aren't enough cybersecurity professionals out there to replace them. Again, if you're not building from source, you're going to have some way of figuring out where the fox is among your binaries.

So that kind of AI assisted binary scanner is kind of the best in class at this point in time. We would recommend not important binarys in the first place and using source code instead. So if you build from source code, it's a much easier way of detecting those boxes Now to give AI its credit if it is trained correctly and you review it, don't just trust it.

It can hallucinate and everything. I think it can, it absolutely help a lot of your development tasks. I'm just a little bit afraid.

A lot of people are like, oh I can just have the AI do it for me and I don't need to double check or I can use a generic ai, I don't need to use one that was trained specifically for my use case. So that's a little bit concerning. So I absolutely think it can be helpful but uh, to your point, there are more efficient ways of doing it, which is actually building from source instead of just trusting.

I'm getting this and I'm going to believe it is what it says it is. But going back to burning out the infotech people in a agile environment today, we're always talking about doing patches and doing them regularly and I believe that over 70% of your vulnerabilities that are found in your code can be resolved through a patch or an update. Yeah, it's hung by a huge problem in the industry, right?

We always focus on the next shiny object and forget to maintain the existing code that we have. And that's really comes down to an opportunity cost problem, right? So if I'm the product manager and I have the ability to close a future function gap with a competitor or I could maybe look at my list of vulnerabilities that I should probably be patching, I'm kind of thinking like could get hack, but I am gonna miss out on revenue so I'd rather go for that revenue, right?

I'm measuring the market share so I wanna be able to close that feature function gap. That's where I'm gonna put my resources, that's where I'm gonna concentrate my development efforts and it would be nice to have that patch and maybe I can squeeze it in next time and then of course that becomes next time and next time. And so we never get around to this and I think the bar is 60 days is kind of the window right now to actually address these things.

But once you've addressed it, you still need to roll it out. It's okay if you've got a sash application, it's fairly easy to roll that out. But if you've got customers that need to adopt it and that can get even harder pushing that downstream.

'cause customers aren't always gonna adopt things at the rate that you wish they could adopt. So it is a problem, it has been a problem forever. And so you know, if my customers aren't gonna adopt it, why am I spending all these cycles actually patching this stuff out?

So it becomes really a cost benefit analysis and it tends to lose out too many times, Which is bad, but true. I think there are plenty of statistics saying that people are running old code buggy code, vulnerable code in production for that reason. And I'm sure there's also other reasons why people, maybe they're afraid of what kind of breaking changes are gonna be in there, which actually gives us a really good opportunity to talk about what active state done and how we can help you.

So like you said, if your area of expertise is not building that source code and then integrating it in why should you be focusing on that maintenance? Why should you be focusing on all of that work that maybe is not as exciting to your developers when there are companies like active state that you can outsource that to and we'll take care of that for you and you can focus on the things that are the new features that are gonna bring in revenue or the new things that are gonna make your system more performant and speedy or they view money on your CPU cycles than memory. So that is certainly a cost value you could do that is not gonna force you to push off your developer's time.

It may cost you a little bit of money, but you're gonna save money and hopefully make more money releasing those new features. So kind of getting into that, like what would you think is kind of the coolest benefit that comes from being an open source management platform for an enterprise? Like what is the coolest thing you think we bring to enterprise or Yeah, it takes away that whole opportunity cost right now.

I don't need to do the trade off. If somebody's gonna take care of that for me, I can focus on my proprietary code and leave all of that, uh, open source stuff, which is basically just commoditized stuff. So be, go back to the seventies and eighties in the manufacturing world where uh, manufacturers used to manage their own supply chains.

They eventually realized that the model was actually a drag on production. And that's what really we're talking about here. By outsourcing that supply chain, you realize immediately the efficiency that you can gain by having somebody else deliver the parts that you need just in time for your manufacturing process.

And that means you can take all of those components and remove them off of your plate and just finish work on that finished product. We can provide those open source components that you are, uh, going to need in your software development process. We're gonna look build them just in time for you on demand, deliver them to you.

You don't have to work with them, you don't have to manage them. We're gonna build them securely from source, from embedded source code and we're going to be able to provide that to you so that you can just focus on your proprietary code that binds together those commoditized components. I mean, these things are commodity components, so your competitors have as much access to them as you do.

Uh, there's no way to differentiate your offering if it's all just based on open source your competitors have access to. It's really that proprietary code that's going to provide you the differentiation you need to compete in the marketplace. So that's what we're freeing you up to do.

If you got to spend cycles patching it instead of actively working on your proprietary code, that's your opportunity cost. If you let us do it instead. In other words, we'll identify what the critical vulnerability needs to be patched for you.

We'll find the next version when it's released, we'll rebuild your runtime for you. Uh, we'll look at it to make sure if you give us access to your GitHub repo and your test suite, we'll make sure that we can, uh, patch that proprietary code as well. Fix any breaking changes, test it to make sure it's valid and give you a branch.

You can take it any time. Uh, so you can continue to focus on your proprietary code, closing those competitive gaps and earning revenue. Absolutely.

So what you just described is what we're billing as our get current stay current feature set, uh, which is just different pieces of our platform that together get you current, keep you current. Like you said, there's a branch. It automatically handles all those breaking changes.

And uh, I mentioned AI a little bit earlier, we've got ai, but we've also got the 20 years of expertise in Python. So we're not just trusting magic, we actually have real people saying, okay, no, this the good recommendation. I think the other cool things that come along with this, as you said, we built everything from source and we have, uh, thought the level three environment.

And so you can get those SSA attestations, you can get a full et bomb that includes both your build time dependencies and your runtime dependencies. We have, you know, a list of all your vulnerabilities if your security team wants that. We've got a list of all your life and Bill in case you've got an open source management team internally who wants that.

And top that all off with our basics, which are portable developer environments. So you can quickly install anywhere and uh, switch around between different projects because a lot of people aren't just working on one project, they're working on multiple. So like you said, let's get you to focus on what really matters, which you know, is gonna be taking a look at where your risk is.

Are you using a package that is not maintained by the community anymore because it's not just about vulnerability that's, has this one fallen out of support of the community, has this one then abandoned? Then if you can just hit a button and instantly get it, that makes that a little bit easier. The whole idea of safeguarding the open source ecosystem, I mean, sitting in your enterprise, it's hard to be able to reach back into that open source ecosystem and be able to create some kind of a solution at that end.

You really have to bring this into your own organization, these kinds of best practices to make sure that the ecosystem that you are exploiting, that you are using within your uh, own, uh, application is going to be secure once it crosses the threshold of your organization. Now we always encourage people to contribute back to the open source community. That is absolutely one way that you can help, uh, safeguard the open source ecosystem, participate in it, bring your own best practices, your own best security practices back into those organizations, work with them, help them understand the kinds of problems that you experience day to day working with these kinds of things so that you can help push forward their ecosystem and make it easier, faster, cheaper, more secure for everybody who leverages it.

And you don't have to be a developer to do that. You can contribute your time as a QA tester. They need unit tests.

You can contribute time as a tech writer. They need documentation. If, like we said, everyone has more than half their code is open source, what is one of those projects that you feel you could give a little bit of time?

Or if your company has money, a little bit of money to help that project do better and be in a better position. So it's less likely to have issues, you know, help them have best practices. Um, but with that, I think we kind of covered the major points unless there was anything else you think we missed in there.

I hope that people take away from this that it is possible to secure your open source. It may not be the thing that your company is focused on and it may be a better idea to give it to people who are actually in the business of doing open source management for you. That's certainly what active state is here for.

com. com. Yeah, I think that pretty much miss everything.

We can do that for you. You obviously can do it yourself so you have that time. But no matter what it's be able to see what you've got and make it easy on your developers to update things.

Make it easy on your customers to update things and contribute to the open source market.