The Rise of Malware: Securing Your Software Supply Chain – From the Source EP1

Hi everyone, and welcome to the first ever episode of our new series, uh, from The Source. I'm your presenter, uh, Ike Turn and Field, CTO, uh, here at Sonotype, and our other presenter. Hi, I am Brian Fox, co-founder, CTO at, at Sonotype.

So, uh, from The Source is intended to be a biweekly, uh, show where we'll, uh, focus on our, uh, special domain, uh, software, the area of software supply chain management, and all the intricacies, uh, intricacies of that. In each episode, what we're going to do is dive into key topics. That includes the latest regulatory updates, best practices for maintaining, uh, open source compliance, as well as emerging threats, uh, post my malware, and anything really that kind of affects, uh, the life of developers, organizations, DevSecOps, professionals, uh, when it comes to our, uh, software supply chain management, uh, overall.

So, uh, Brian, I'm super excited. Yeah, likewise. I think, um, there's, there's, there's a lot to learn in this space.

So you had some examples of some recent, uh, malicious types of attacks that, uh, that, uh, happened this summer. Do you wanna just give the high level on that? Yeah, absolutely.

Um, so what's been happening, uh, happening over on the, uh, JavaScript ecosystem, MPN is that over the summer we've observed huge amount of, uh, spam entering that ecosystem. And what's behind that Skip spam is actually really, really interesting. So, so these are essentially packages, uh, that have been designed to look like real open source.

Um, they're downloaded by each other. There's, they're networks that also stimulate the downloads. And when you crack open the lid and you start looking at these packages, they're kind of nonsensical names.

They're kind of, uh, kind of, uh, you know, published in an automated and sort of, uh, flood like fashion. You know, thousands of them pop up every single day. Um, what ends up, uh, ends up actually being behind it, is that, uh, is that, um, uh, somewhere along the, uh, year, there was a new pro crypto protocol that was announced called t uh, do XY z, uh, to be precise.

And it's called some pretty legitimate backers behind it. Actually, one of the people that invented Home Brew, uh, if you're a Mac user, um, you know, many of us, uh, have used that, uh, package management system. And, um, what they did was, uh, they published a protocol that aims to reward open source contributors.

They aim to aim to, essentially, every time you publish an open source package, you introduce a little token in it. And, um, and that, that token is monitored by sort of the crypto network. And, you know, the intention was that if your package becomes really popular, then you get rewarded with a little bit of cryptocurrency.

And the idea is, Hey, you know, we should compensate our open source maintainers better. Now, of course, this being the realm of software, we can't have nice things. So pretty soon after that, uh, was, uh, published, um, you know, bad actors basically overtook that, uh, logic and started just publishing outright spam and then manipulating the targets, uh, manipulating the, uh, downloads of those packages altogether.

So basically they, they, uh, download them from the internet, uh, kind of overinflate the numbers, and that obviously stimulates paybacks, uh, for the developers. So, over the summer, I think, you know, Brian gave me honest here, we've seen, uh, you know, hundreds of thousands of these sort of appear. And I think I read from somewhere that actually most of the newly published practices over the summer in NPM were basically just bam, you know, that was related to this sort of, uh, the concentrated campaign.

Yeah. So I think the, the, the, the fact that this is happening highlights a, a, a major flaw in the, the supply chain. Like you said, the motivation in this particular case was for somebody to, to game the system and get some more payout, to make a bunch of pretend packages look like they're really popular, more popular than real things, and therefore basically steal effectively some of these donations.

But I think in order to really understand how we got here, I think we have to take a, a, a bunch of steps of that. Um, you know, so if, if we, if we think about how software is developed these days, right? It, you know, when I started, uh, a long time ago, you know, we sat down and see, um, and, uh, and, and, you know, started at void, main void and, you know, wrote all of our code from there, um, open source came along and, and early days, a lot of the open source pieces, especially if you were looking to include capabilities into software, you're developing, not not open source as in like Firefox or Apache, HCBD, where you just wanted to use the software, but like you actually needed a class file or some functionality Yeah.

Like curses to, uh, highlight your terminal or something. Yeah, exactly. And so in those days, you basically had to like, go get the, the source code and you like, would copy and paste it into your software and then massage it and make it work, right?

So you were like literally taking a paragraph from a book and putting it into your book. That's how Open Source largely started in terms of like leveraging the capabilities. Fast forward a little bit, you know, we got to the point where you had whole reusable modules, um, in Java, these would be the jars, right?

In, in Windows C world, they were lib files, shared objects, DLLs. And so these were whole compiled pieces of units. So sticking with the book metaphor, it's like taking a whole chapter and including it wholesale into, into your software, right?

And so this made it a lot easier. You didn't have to just cut and paste code. And also, um, it was easier as those libraries matured and fixed bugs.

You just picked up that whole thing and included it. You didn't have to rework the custom code. Um, and then package managers came along, you know, CPAN for Pearl was a very early one, but then also Maven, uh, one that I've been involved in for, what, 21 years.

Wow. It's a lot weird to say that. Um, um, Apache Maven is a, a very popular build system in Java that really modernized some of this dependency management, um, S space, basically.

And so what happened there was we were able to publish these open source components into a repository on the internet, along with the metadata that included information about the component, but also that component has dependencies too, right? Yeah. So these are transitive dependencies.

And so these repositories in, in the Maitland, this is what we call Maven Central, or the central Pository. Turns out Sonatype runs this repository, but other ecosystems followed suit. You have, um, the pi pi, you have the new get gallery, you have NPM js, um, Ruby, and you know, Docker And you know, yeah, right.

D Noland and all, all of these other places. Yeah, That's right. These are, these are the public registries where people can put and share these components.

Now, the thing to understand is that most of the interaction with these, these tools, um, you know, you, you add the dependency to your build file somewhere, and the, the tool goes off and it fetches this automatically from the repository. And then in many of these cases, it reads the metadata and says, oh, I, I have that thing. I have to get these 15 other things.

Um, and it will do this automatically. Some of those ecosystems, um, that process is actually executing code that's been fetched because they need to be quote unquote installed. You know, RPM is a very well understood thing.

It's a very similar thing to RPM where it's actually installing the software. Yeah. And in, in security terms, that's, uh, by definition a remote code execution, right?

So I think you can start to see where this, this challenge comes from that, you know, did, did you wanna add anything that Okay. Yeah, I mean, I mean, that's exactly it, isn't it? It's, um, it's, you know, by design, every single modern software built manager, which is in every single programming language has a, is built in remote code execution flow.

And by design, I really mean, yeah, in most cases, you need to compile, you know, the library to the right environment. You might need to fed some extra components. You might need to do some massaging, especially, you know, in these sort of AI projects, you often see the libraries are very sensitive to the target environment.

You need to have certain drivers and, and things like this. So it all, uh, all kind of works together. And that in on itself is pretty dangerous because there is usually no developer interaction.

There is no, hey, developer, here's the lines of things that are executing. Even if that was printed out, you know, you could certainly put a minus X or something like that on the command com command line that makes it sort of, uh, very verbose. But the issue really is, uh, there's fairly little, you know, that you can do.

Once the package installation starts, it will compete, and we assume that the package does legitimate things. Now, sort of another fundamental weakness, uh, to add on to that, that exists sort of within the software supply chain, when we talk about dependency management is in the upstream, in these registries, um, cent is a bit of an exception. We kind of take a slightly different philosophy.

I'm, I'm, I'm sure you know, the, um, uh, people, uh, people, uh, who like to, uh, comment about that will, will, will forever have a different opinion. But, um, in, um, places like NPM and Python, you know, in many open street systems, there's sort of a philosophy of, uh, you know, we have open access. Anyone's free to register.

It's done on a fairly, you know, good intended philosophical background. It, it, it promotes, uh, a contribution back makes it easy, lowers the barrier of energy. So anyone that's kind of, kind of, uh, can publish a package, it's very easy to do so.

But the flip side is, in many places, there is no enforcement or control over the namespace of those packages. So that leaves us open to this sort of field of, uh, uh, this sort of one, uh, uh, risk landscape of people, for example, finding a very popular package and then registering a, a sort of typos coded version of it, like changing one letter in the package name. Uh, so instead of requests, which is relat relatively popular, uh, Python package, there's a legitimate malware campaign, uh, that wa was called, uh, using a package name called request.

So publish a couple of versions that make it, you know, fake the read me. It looks perfectly legitimate. And sort of that's combined with the fact that the way that developers actually install these packages in 99% of the cases is they have to type the name out.

It's like typing a name, an ad address onto a postcard, right? So you do MPM install requests or PIP install or, or, or whatever, uh, or MBN install. And, and so when a infinite amount, you know, there's tens of millions of developers across the world, they do this sort of, uh, activity on a daily, uh, basis, multiple times a day as well.

Eventually, they are going to make a typo. Uh, and that's really what these two methods of attacks, uh, rely on, you know, in many cases when they ruin the, using phishing tactics. So in some ways it's not very different to phishing.

We were kind of talking about this in the pre-show, but, um, you know, sort of all the same gamut, uh, types of vectors, uh, appear, right? People can do targeted phishing. They can use spear phishing against a single person or a single company, or they can go broad base and use it, use it to, um, man to, uh, distribute malware.

And they've really realized that it's quite an effective method of, uh, releasing malware. So, so, uh, Brian, we actually kind of keep track of, uh, what we're kind of seeing. So o over the last few years, I mean, there's been pretty phenomenal growth in these, uh, types of, uh, attacks as we've seen them.

Yeah. Some of the first ones that I, I picked up on and started, you know, on the conference circuit talking about this problem was way back in 2017. Um, and, and you could very quickly see the evolution of the attack vectors as each subsequent, uh, attack kind of built on the things of the, the ones before it.

But, um, you know, last year we were at 250,000 known malicious components. This year it's over 700,000. So it continues to double or more every year.

We haven't hit that law of large numbers where the percent of growth tapers off yet. And, and, you know, I take this as evidence that this is working. You know, it, it's, um, it's working.

That's why the attackers keep doing it. Um, some of these attacks are quite significant. I think we're gonna get into some of the, the different types of attack in a minute.

But, you know, it, it, um, the reason why, again, that this is happening is that the industry at large does not have good enough defenses against this. And, you know, we, I've been exploring different metaphors. You know, if you think about food at a picnic that's left out too long, that gets maybe a little bit bad.

You could think of that like a vulnerability. It might be okay, it might hurt some people who are immunocompromised, but most people are okay, that's like your typical vulnerability. But poisoned food is very different, right?

And I think these intentionally malicious components are more like poisoned food. And so the, the challenge is that organizations try to rationalize this away. They try to handle, uh, all of the dependency management problem the same.

And they have this mental model of how we deal with vulnerabilities. We prioritize them. We try to fix the most important ones, and, and we'll, we'll ship the release.

And, you know, it's a big pile. We can't fix them all, right? These are all va very valid, um, rationalizations in the, the, uh, vulnerability world.

But when you're dealing with intentionally poisoned food, the fix is not go put it back in the fridge. The fix is not eat around the mold. Um, you know, and, and I think that's the part that's missing, is that so many of these attacks are, are perpetrated and they execute as soon as the developer downloads 'em.

So by the time you put it in your list and you think about prioritizing the vulnerability, you might have had tens or hundreds or thousands of machines actually, you know, having the payload execute. And we've seen this happen with, with large, um, you know, financial institutions that turned this on and started looking for these things and found that they were already, um, you know, baked into applications and deployed all over their, their, their portfolio. Um, and so this is a very real problem that not enough people are thinking about.

And so, Ilko, you gave some examples early on. You know, these were, they, they were sort of not super scary from a, from an end user perspective 'cause they were trying to really just steal the crypto game, that popularity system. But let's talk about some of the other types, uh, of very real attacks that we've seen.

Yeah, I mean, um, I mean, uh, there's been sort of a huge evolution of, uh, the types of malware that we're, we're, we're seeing, right? Um, and, you know, it kind of follows through, um, almost mirrors the popularity of open source itself. You know, as, as it's become more adopted, more ingrained, it's kind of given motivation.

And so, you know, kind of worse and worse types of, uh, creepy crawlies have started to appear. So to your point, like very initially onwards, uh, initially in the beginning, the types of malware that you'd see distributed tended to be things like discourse session dealer. So it was, it was, uh, sort of people that were just trying to steal an identity often.

Uh, we started seeing crypto miners. Those were sort of the early, uh, supply chain attacks that you, you start, we started observing where, for example, you know, you saw, you saw people distributing just, you know, making it, the package look like it's legitimate, adding crypto miner in it, uh, developer machines and highly privileged, uh, uh, hardware, uh, usually run on RAM and CPU. So, uh, nobody's usually none the wiser if you get into a CI pipeline.

There was at least one heist that we, we saw that, uh, uh, used, uh, a, uh, uh, vulnerability, a vulnerability to poison, um, uh, shrinkage machines. And then, you know, mine crypto essentially on those builds, uh, build agents. So, so, um, you know, initially it was that sort of thing, you know, dangerous, yes.

Criminal, absolutely, but not necessarily sort of, uh, dangerous from us or resilience and security, you know, organizational security perspective. Over time, though, it started evolving and we started seeing sort of the more serious, uh, gangs appear. You know, somewhere around, I'm, I'm gonna say 20 18, 20 19, we saw the first orchestrated targeted crypto heist that used poisoned, uh, packages pretending to be open source.

And what they did was, uh, well, actually this was a takeover attack. They, they took over an open source project, and, uh, they knew that that open source project was adopted by a specific crypto wallet. So they poisoned that project, and then when it was running in that crypto wallet, they actually stole the funds.

And I think that group made with, made a way with like millions that, Yeah, and just to put a fine point, I believe this was event stream, and it was 2018, I think, and copay was the target. But the important point here is not so much that it was crypto, I mean, it was important for them. This attack was targeted and delivered a patch file that patched custom code inside of the copay website, right?

Yeah. So that could be anybody's application. If you imagine that these, these components can apply a patch to your custom code, well then it's game on, right?

And, and that's the real takeaway, I think, from that event stream one. Um, and, and again, this was 2018 when, when this happened, and, and, you know, we were out there talking about it, that was a long time ago. Yeah.

Um, but we've seen them all, we've seen it, you know, stealing AWS keys from machines and databases. Um, you know, we've seen some attacks that got in that, you know, got into, um, you know, camera, uh, streams and things like that at hospitals and police departments and, and things like this, right? All of these through these, these dependency, um, types of attacks And, and probably pro.

And so you might be thinking, Hey, nothing new under the sun. It's all malware that we deal with, uh, through all the other ecosystems. Well, what makes these things particularly dangerous, A, is that they're running on privileged hardware that usually have privileged access.

Uh, and b, it can affect your own code, which means that you, you know, they're often used as a starting point for more serious multi-phase, or what we like to call 'em, sophisticated operations, you know, when you, when you see the press releases, right? Um, and so probably the most serious ones that we've seen are legitimate, uh, legitimate pieces of malware. For example, COBOL Strike, we found some packages distributing COBOL strike a couple of years ago, which I think was sort of initial warning bell, uh, that, you know, sort of the big serious groups have moved into this territory.

And more recently we caught, uh, some packages actually distributing lama, which is a known malware strain. Uh, you know, it's, it's thought to originate from sort of Russian nation state, uh, originating hackers. Of course, these tools, once they spread out, they, they get used by other, uh, groups as well.

But, you know, that's what we associated with mentally. Um, and kind of interestingly, uh, wma, the malware family was amongst the families implicated by Mandiant when they investigated the recent Snowflake breach. So if it is actually a really interesting read, we'll, we'll, we'll add that to the show notes here, but, um, uh, the Mandiant, uh, uh, Mandiant report on the Snowflake breach kind of talks about how they, uh, use the, uh, how they kind of got, uh, hold of snowflake's authentication tokens, and then use those to hack their customers or kind of hack into the instances and then, you know, do what they did.

And it was found that they, they heist started by them distributing info steering malware, and one of those strains was Lama, we've seen that same strain happening, uh, being distributed in the supply chain. It kind of makes sense, right? You know, it's, it's something that, you know, developers would have a hold of, uh, hold of as well.

So un uh, unfortunately, this type of more serious, well-funded and well motivated thing, uh, has moved, uh, for good, uh, into sort of the threat landscape that now faces us as developers and unfortunately, fairly on targeted. I think a lot of these are sort of cast out net types attacks where they do try to typo what something that's very popular, and then they should the malware. Uh, and, you know, the beauty of it is, uh, for the hackers is, you know, if it hits the developer machine, it is executed.

It's almost guaranteed. Yeah. And one, one thing, like, while some of those examples you gave are, I, I would struggle to define this classic malware that might be picked up by your typical, you know, uh, endpoint systems and malware defense systems, things like that.

Many of them aren't, many of them are custom code that is designed to run in the context of the build. So it's not like there's a known backdoor or root kit that your virus tool or whatever tool is gonna pick up and see. That will happen in some cases.

But what we're seeing is most of these actually don't look like that. They look like open source custom code that's doing something very fishy. Um, and, and, um, and so traditional malware defense systems are not catching up to this type of problem.

And, and as evidenced by the explosion of them, we're not solving it, right? And so this is something of course, you know, I was out on the conference track in 2017 highlighting this problem. Of course, we've been working on, on solutions to this, right?

And, and this is why we're raising the awareness. 'cause we see how many of these, these packages are, we've, we've gotten really good at identifying them. We have ways of blocking them, um, at the package registry.

Now, when your developers are actually downloading these components, and, and that's really the only way to stop it, you have to stop it before it executes, and again, it's executing as soon as they download it. So scanning before you release the software completely misses the whole, whole war, right? Yeah.

I think that's, that's a really important call out actually in the context of malware, right? Um, if it hits the developer machine or any tool, uh, that tries to run an automated build is game over, their prevention is the only cure. So, you know, in terms of, uh, in terms of, uh, and when you add on to that, that you're certain regular, uh, endpoint security tools just aren't enough, they don't have this sort of data.

It's often custom code, it's deployed once it lives for a couple of days, then it's removed, uh, from the world. They just, uh, can't keep up with that sort of volume. So what's pretty interesting is we've kind of inadvertently, um, you know, become one of the sort of leading authorities, uh, in helping our customers prevent this.

And we do it by the means of, for example, auditing or their open source package, uh, traffic as it comes in, and then looking for sort of signs of compromise, uh, as well as known, uh, malicious employee packages. And I think, um, at this stage, we're like discovering them, you know, within 10 minutes of them being published. But, you know, sort of technically speaking, if you're thinking about ways of defending this, I think like the couple of takeaways that, uh, folks ought to take out of this is a, it's a real hole, and it's already there.

It's being very actively exploited. And two, prevention is the only cure. Normally people try to, especially in the context of vulnerability and DevSecOps, we try to avoid building breaking builds as much as possible.

Everybody's sick of gates, but this is the only place where legitimately stopping a download from completing. And so breaking a build sometimes in a fairly violent fa fashion is the only way to prevent the payload actually from executing. Unfortunately, these payloads are, you know, so, so numerous.

We've actually collected a timeline of all of these packages that we've seen sort of, that kind of highlights. We'll make sure to put that on the show notes, but it is a very long laundry list, and unfortunately, it's getting worse. Yeah.

The, the numbers are getting so big, it's hard to even talk about all of them anymore, which is why we have to move to broad classes of, of problems now. Well, uh, that's, that's the evolution. So, uh, we're, we're, we're, uh, about to run out of time, uh, for, uh, this first episode.

Uh, but, uh, Brian, um, it's been an absolute joy, uh, to our deep dive into this topic. Any sort of takeaways, uh, takeaways that people ought to take aside from everything we've spoken about, All the things. Um, importantly, you can't, you, you, you can't treat these malicious components like you are your vulnerabilities.

They're basic, are, um, very few ways of solving this. Most organizations have none, right? And so if you think you have a solution for this, I, I bet you do not.

Um, and, and I see this time and time again, and so people are willingly leaving their, you know, their front doors open, um, with the bad guys, you know, trying to come in, um, thinking that they have defense. And, and that's the real problem. That's why we're trying to educate people that this is a different problem, requires a different solution that you probably don't already have.

Well, uh, on that depressing bombshell, Brian, I think we're gonna, uh, gonna call it here, uh, this time. Uh, thanks very much for joining us. Uh, tune in, uh, in a couple of weeks time, uh, for the next episode, uh, of, uh, from the Source.