Episode 1: The Rise of Software Supply Chain Attacks – ActiveState Podcast

Hello everyone. Welcome to this episode of the ActiveState podcast series. This is episode one, the Rise of Software Supply Chain Attacks.

I'm Pete Garon, I'm director of product here, and joining me is Dana Crane. I am the product marketing manager here at Active State. Welcome everyone.

Yeah. And today we're gonna talk about software supply chain attack. Maybe before we get started though, we should talk a little bit about just what active state does here.

Active state. We produce an open source management platform that makes you more secure and productive. Yeah, we like to say that it helps you integrate software at scale.

To do that securely, you're gonna have to manage your supply chain and that supply chain is gonna consist of a lot of open source. So that's what we're here to help you do. And so maybe when we're talking about supply chains, maybe we should just get into it here.

What do we think? How would you define what's the software supply chain do? So the way I kind of think of it is you've got sort of three parts to it.

You've got the import process where everything comes in from the ecosystem. That can be multiple ecosystems 'cause you're working with multiple different languages. Then you've got to incorporate that in your software and build it in some ma manner, usually through a, a build service.

And then finally you've got that deployment service. So how do you hand out those packages to the right people to get the software built? Or maybe it even goes further downstream out to your customers.

So that sort of import, build, deploy that kind of three-legged stool. That's why I think about the entire software supply chain. Yeah, I think that makes sense.

I think most people are familiar with what supply chains are for a regular product, right? You get your parts from here, you get your, you assemble it in a factory, you ship it out kind of thing. But I think that people maybe don't even think about software supply chains with software.

They're just like, oh, well I just magically get it from the internet. The software fairies deliver it to me. Yeah.

I think there's a, a number of startups that don't have much process in place, so they're willing to let the developer download, install it directly from the internet. Uh, but there are a lot of customers that we have on our end that have a lot more processes in place. They wanna make sure that software is actually fit for use before they hand it out to their, uh, developers to get started on the software they're building.

Well, and that's where, that's a really good example that you bring up there where I just go out to the internet and get my software, but you don't know where that's coming from, right? You don't really have any information about what you're, there's a lot of implicit trust going on there in terms of what you're installing. And there's a lot more opportunities I think for there between those various stages, right.

For attacks to happen. What are some of the attacks that can happen on these di various repositories? We, you know, Yeah.

I mean, when we think about the open source supply chain, we're talking about an open ecosystem, right? Which is almost saying like, it's vulnerable by design. You need to make it as open as possible so you get enough authors to come and contribute their free time to make your ecosystem viable.

You don't wanna put a lot of security controls in place that could sort of lock them out or make it a much harder experience. So we kind of talk about it as vulnerable by design, but on your end, when you bring it into your enterprise, that's when you have to start locking things down. And there are so many vectors of attacks these days.

Everything from type of squatting to malware to even GitHub fork libraries out there. And it's not just on the import routine. When you start building, if you're build services and secure, we've seen that back in 2020, the end of 2020 when SolarWinds got hacked and they managed to insert a piece of malware before the signing step and then deploy that out to all their customers.

Everybody installed it, sign software, it should be fine. And it turned out not to be fine. So you've gotta really think about all those different vectors of attacks that are now becoming very prevalent in the su in the supply chain.

Well, when you think about, uh, a lot of the more recent attacks or noteworthy kind of events, cybersecurity events, almost all of them are supply chain attacks. It's Vulnerabilities Are still a thing and they're still, they're still out there, but a lot of these are attacks where they're attacking the supply chain and they're doing things like that are from the very basic things like the type of squatting and malware where type of squatting where name something, one letter off from some popular package and hope that somebody accidentally types it in. But there's also pretty sophisticated ones like the most recent XE one kind of thing.

Yeah. Where, You know, somebody who was probably pretty well funded, played the long game and got, became a contributor on a open source package and had a, had, you know what I mean, prs that were injecting back doors and sock puppet accounts and things like that, that were all part of this like orchestration to inject very Sophisticated, Inject something. But this is like, this is much different than, oh, there was a bug that allows a buffer overflow that could be exploited or something like that.

This is, this requires a completely different level of protection, I think to even be able to deal with threats like this. Like you're saying about like injecting things into the build process, injecting things, you know what I mean, into the deployment process. There's all kinds of, all kinds of different vectors there.

Yeah. I I I think that SolarWinds hack was a really good poster child for this kind of thing, right? When suddenly all the bad actors in the world wake up and realize, geez, if I just infect one popular piece of software, I can get into multiple US government departments.

I can get on multiple military arms of the government. I can get into all the telecoms, I can get into all the accountancy departments across the world. This is a great thing, man.

It's like you're leveraging this economy of scale just by hacking one popular application. You can get deployed downstream to tens of thousands, maybe even hundreds of thousands customers. You don't have to go one by one sort of looking for holes in the network or being, if there's a zero day that you can penetrate one by one here, you've got a hundred thousand potential victims right.

At your fingertips with just one single hack. Yeah. Well, and I mean the, the obviously in some ways the poster child for that is like the log four J thing that Oh, right.

Happened right where share it was, there was a vulnerability in there. The, the, the reality was nobody un no one knew how widespread this thing was, right? That like a hundred layers d people are still finding that, oh yeah, this is in my software and it was infecting basically everything.

Yeah, that's, that's another good one. Uh, the other thing to think about is it's not just coming from the public repositories. That's typically where you see, uh, a lot of the bad actors uploading something bad, some piece of malware or something into the public repository.

They're all prebuilt of course. So all these prebuilt packages, they're binaries typically, and you're down building them to your system. So you really can't see what's inside of it.

There's no source code to look at. It's all binary, so it's all obfuscated, but they have to compromise, right? Like you were saying, they have to typo, squat, add an S to the end of the name of a package and hope somebody fat fingers that not a great chance of success.

Uh, but we've also seen them go back upstream. So not just going to the public repositories, but going upstream to the actual repos where the source code exists, such as GitHub and trying to clone those, um, repositories, make it under their own name. So now you can have the actual name of the package, it's under your name, but it's the name of the package.

And now you can go out and social engineer and try and convince people that you have patched some critical vulnerability or you fix the bug and yours is the most latest and it's the best one to use. And that's a great way to infect you as well. So things need to be careful of it there.

Yeah. Yeah. There's a lot, there's a lot.

It's have these been around forever? Is this well, why all of a sudden this is all we're talking about? Yeah.

Driving that. Do you think That's the weird thing, right? They have been around forever and type of squads been around a long public repositories that been around, everybody's trying to fish these sort of things, right?

But why the concerted effort, in my opinion, is because of the pandemic. Everybody moved from the office to unsecured networks at home. Some people were v ping in, some people were just using the local network.

All those kinds of defense in depth things that you might have in a more structured office setting had gone away when you moved back home into your office home, your home office, which doesn't have that same level of security around it. I think a lot of bad actors sort of perked up at that point and said, wow, this is a great opportunity. And since many of us IT workers are still working from home, uh, that opportunity still exists.

A lot of us have gotten better at home office networking, but that opportunity still exists. I Think that also it's has to do with like open source usage is growing exponentially and the rise of GitHub and the proliferation of that kind of thing, like the popularity of all of these things like NPM and Python pipi and those things are all kind of some ways have much higher profiles than they used to. And so I, I think we're past probably the heel of that exponential growth curve for open source adoption.

You know, some huge percentage of all applications contain open source. I, I, I don't know the exact number. I think it's over 90%, I'm sure.

And So, yeah. And so we've basically hit this critical saturation point where now that's the, if you want to get into, that's the, the primary vector to get into basically any piece of software around the World. So yeah.

And, and when you think about it, most software these days, 80% are better, is composed of just open source components. And that's what really drives the sort of model of software production these days. Back in the day, everything was proprietary.

You wrote all your code. Now it's mostly just counting on other people's code. So bringing it in, and we like to think of that sort of as a software factory.

And really all these packages are like your supply chain, right? So you're bringing in all these packages, you're assembling them in different ways, and you're gluing them together with your own code making, uh, you know, your own proprietary software that you sell. But despite the fact that it's 80% of the product that you sell, it's not what you actually sell.

So this whole software supply chain thing is sucking up a lot of resources. It's sucking up a lot of time. It's sucking up a lot of effort.

You've gotta be able to build out that software supply chain in a way that makes sense for your, the way that you operate, the way that you, your software development process work, but it's not the thing that you actually sell. So it's not the focus of your business. And so when you do things like maybe use two or three different languages and each of those languages is gonna have a proprietary way of doing different things to do with their ecosystem, but you can't create a solution for each of those three different languages that you use in your business.

That would be way too costly. So you put in place one sort of good enough way of doing that supply chain to kind of stretch it to accommodate the other couple of other ecosystems that you're using. And it doesn't quite stretch there.

And so you end up with a good solution that doesn't quite get you the productivity you were hoping for doesn't quite get you the security you were hoping for, but you know, you gotta live with it. What's your take, Pete? Yeah, I think that's, I I think that what happens is, like you said, is that uh, people, you know, you're sort of cobbling together a bunch of different, different solutions to kind of approximate something that's keeping an eye on your supply chain, but probably is the vast majority of things are really focused on vulnerabilities these days, right?

And like you said before, it's not necessarily looking at the things like you said, you're ingesting a lot of binaries on the internet. Was that just made on somebody's laptop in a closet? Is it the actual code that you think you're installing?

There's a lot of, there are a lot of tools that are not looking at even looking at problems like that, right? They're mostly focused on the vulnerability side of things. And so, like you said, it's, you're spending a lot of effort to keep track of all this stuff, right?

There's huge volumes of these things. There's a huge amount of plumbing that you have to do just to be able to manage that stuff at scale. And like you said, that's not their core business, right?

Yeah. That's just, you're just managing the incoming parts for your factory, Right? Right, right.

Yeah. So you think of this back in the seventies and the eighties, you think of the manufacturing model, right? They all used to own their own supply chains used to manage their own supply chains and they, they moved to a just in time model where you pushed it back on the supplier and the supplier would deliver it just in time to be built into the product.

And so the manufacturing class was able to move on from managing the supply chain to actually focusing on their end product instead. And that's the kind of thing we think is where the software industry is going as well. You have to stop managing those components and start focusing better on your end product and basically just doing what the manufacturing did, outsourcing that management of the supply chain, right?

That's what we tend to do here at Active State. We help you manage that supply chain that is our business. So that's where we're focused.

We're trying to be able to help you understand that we can do it more securely. We can manage these things for you. We can take a lot of that stuff off your plate so you can get back that productivity, get back that security assurance that you were hoping for when you're managing it yourself, but never quite achieving.

Yeah, I think that's, that's probably one of the advantages is that you, if you are outsourcing, you can outsource to somebody who's an expert in this versus trying to carve off some small percentage of your developer's time to, to do this when they're probably wanting to be working on whatever your more product is. And assembling this stuff is not easy either, right? Like we have a pretty, pretty sophisticated system that is looking at all of your dependencies.

We're building everything from source that's out there so that you get guaranteed provenance of the things that you're installing. And we have also universal tooling that is, if you're crossing multiple ecosystems, you, you don't have to cobble together different sets of tooling to, to sort of approximate that. So yeah, I think that there's, there are definitely benefits to, to shipping it outta house.

Yeah, I mean you just, when you think about it, right? You, you're managing multiple package managers, managing multiple environment managers. Uh, there's lots of overhead that goes in here.

We talk about typically 10 to 20% of a sprint going into maintaining your software supply chain, you know, uh, updating outdated packages for example, or remediating vulnerabilities that might come in. Those sort of things that act as a drag on your development. Those are the kinds of things that you need to address all of the software supply chain vectors coming in, but that you might not be thinking of even when your build system have in place.

Is it reproducible? Is it hardened? Do you know?

Do you have those kinds of things in place? Most people have different tooling in place. We talked about the fact that most people are ingesting binaries.

So you've got a binary scanner and that binary scanner is gonna be throwing all kinds of false positives, all kinds of alerts. And then you've gotta count on your cybersecurity team. So now your cybersecurity professionals have to investigate all those false positives at the start of any project that's gonna be quite a volume.

And we've seen over the last couple of years, quite a bit of cybersecurity burnout happening. 1 billion. Uh, so it's obvious that there's a lot of attacks happening in the enterprise.

Your cybersecurity people are getting burnt out and that's in a time when there isn't a deep pool out there to hire from. So you're losing the kinds of professionals that were decreasing the risk. All of this is all coming to a head.

This rise in software supply chain attacks is really taking a toll on the enterprise. Uh, and what we're saying is outsourcing that, uh, whole supply chain can really help relieve the stress on your developers and your cybersecurity professionals. And I, I, I think I know the answer to this, but are these supply chain attacks going away?

So times the thing, it doesn't matter what the reference you use, everything seems to be getting worse and worse. I mean, looking at some of the reports from last year, depending on how you look at anywhere from 60 to 90% of companies out there experienced a software supply check software supply chain attack last year. So it's not gonna get any better anytime soon.

No, it, the, all the numbers I've seen is that it's actually growing like exponentially, right? Like it's growing. I think it grew like 600 something percent last year and that's up from the year before, which is even so, and I think that when you look at some of the recent attacks like the XE one, like the, it's pretty, they're getting pretty sophisticated, right?

And so I think that it's gonna be a challenge for those who are kind of part timing it to keep up with how sophisticated these attacks are getting. Yeah, I mean there, there's obviously the state actors out there who are trying to get the sort of long-term embedded stuff or maybe some living off the land kind of attacks where they can hide things much better and have a very long playing that long, long game. Uh, but there's also even something as silly as, you know, ransomware as a service tell us what your target is and we'll send ransomware that their way until they get infected and then you can make your payday.

And it's not just one payday, it's not just unlocking your data, it's basically unlocking the data and then having to pay them again when they advertise that data for sale on the dark web. So there is no win situation here. Uh, it's all risk all the time.

And you've gotta keep in mind that while most professionals understand that software supply chain attacks are definitely a vector and they are growing, they just don't have the time to be able to deal with it, they're being overwhelmed by the stuff that they have to deal with today. I mean, we talk about vulnerabilities, it's really just the tip of the iceberg, but it's the most visible one, right? It's what everybody talks about, what everybody sees.

And there's lots and lots of traditional vendors, traditional software security vendors out there who help you deal with those vulnerabilities. But you're ignoring all the rest of the stuff, the rest of that iceberg, the massive amount of vectors that are all coming to these days, you're just not dealing with them 'cause you haven't got the time. com where you get a free account and you can start trying this out and making your organization more secure.

And we're gonna have a whole series of these to ActiveState podcasts where we're gonna dive into different aspects of software supply chain security. So stay tuned and yeah, good to chat, Dana. No, great chatting with you.

See you next time.