Malicious Packages Report 2023 – Jeanette Sherman, Mend.io

Well, I'm welcome today by Jeanette Sherman. Jeanette is Senior Product Marketing Manager with Men io. Welcome, Jeanette.

Hi there. It's great to talk to you. Good to have you on.

Well, tell us a little bit about yourself and tell us a little bit about Men io. Yeah, certainly. So, um, I'm really the, the person who kind of keeps an eye on the voice of the customer and on, on the market, and understanding what people care about, uh, in cybersecurity.

So men, we really see ourselves as autopilot for AppSec. Um, we are creating, uh, a platform where you can have the best of control and automation for your AppSec. Excellent hot topic application security AppSec.

Good place to be. io site. It's a great report.

Yes, that's right. I've had a chance to read, read it myself. We're doing some work with you, matter of fact, on some kind of follow on research to that.

So, uh, I do highly recommend it. That part wasn't written by us, but it is really great work. Well, uh, tell us a little bit about, um, kind of the thinking behind doing this kind of a special report.

Why are, why are malicious packages such a big issue? Yeah, a lot of why they're such a big issue is because when people think application security, they're thinking CVEs, right? They're thinking about the vulnerability picture and, you know, and, and, and people have started to scan for those vulnerabilities.

Now, if you're an attacker, you always want, you know, kind of the path of least resistance, um, to, to be able to deploy your attack. And as companies start to resist more, I start to scan for vulnerabilities, identify those vulnerabilities. You know, I say that having a vulnerability is kind of like having a busted lock on your back door.

You could have that busted lock and depending on where you live and everything, nobody might discover that for five, 10 years. But a malicious package is sort of like if you said to the burglar, I'm gonna invite you inside. I think you're somebody who legitimately belongs in this house, I'm gonna invite you in and you're gonna have as long to be here as you want.

And so a lot of attackers see that as a, a really promising vector, maybe more promising in a lot of ways than vulnerabilities. And, um, our research team had started to, I identify more and more of these malicious packages. Uh, we realized that it was time to, to start really telling a differentiated story and talking to people about this emerging threat.

Yeah, it's kind of like all the locks on the, on the building or the house or whatever. Any one of those could be a way that an external threat gets through. Yeah.

In this case, they're breaking into the package manager and you're just simply downloading whatever the latest version is or a new package into your dev environment. So it is sort of a, an indirect, but then once, once inside, of course, now it's a direct threat to you. Exactly.

It's interesting. Um, you know, because so much of our software too is not, it's not all developed by, by us, right? There's different reports on what percentages or how much is actually our code, how much has been open source, third parties Exactly.

Library services, all of those kind of things. And they can be susceptible to Exactly. It's not just your, it's not just your direct dependencies, it's your transitive dependencies too, right?

Uh, if one of these malicious packages gets, gets into any of your code, it can do a lot of different things. We, we found, uh, in our research, uh, some of these packages were exfiltrating user data. Uh, and, you know, depending on what industry you're in, this could pose a, a true existential risk to a company.

Interesting. Well, we'll dive into the report a little bit more. The, the research that you did.

Love to hear some of your thoughts on what some of the things that you discovered, you know, what things are kind of increasing in the growth of mil malicious packages. Yeah, so one of the, the biggest, uh, things that we saw is that just the number of malicious packages is, is swelling, uh, kind of unbelievably fast. We, we saw a 315% increase in malicious packages year on year from 2021 to 2022.

And the big takeaway, I feel like from something like that is if what you've been doing up until now about malicious packages is saying, you know what? This isn't that big of a problem. We don't need to have a response to this threat.

It's so small. You're not gonna be able to keep getting away with that. You know, uh, a year or two more of growth that looks like this.

And all of a sudden that's what you're, you know, you're going to find these malicious packages have already gotten into your code and are doing, who knows what. Um, another of the, the kind of surprising things that we discovered, um, when we turned on a new product functionality that could detect, uh, malicious packages that were already in legacy existing code bases. So, you know, uh, code that had been developed without using any kind of detection tool for malicious packages, when that functionality was turned on, we discovered that there were thousands of these packages already inside, you know, people's, people's code that had no idea.

You know, we were sending, we were sending, uh, emails out immediately to customers letting them know that, you know, kind of the, the invader is already in your house Where the proverbial camel's nose is under the tent. Matter of fact, a good part of the camel is already under the tent. Yes, yes.

Much more than the nose. It's, it's, it's pretty amazing. And that kind of a number 315% does get your attention, which just shows you, I don't know if we're increasingly using packages as well, but that is a major threat factor for us.

Yeah, absolutely. And, and packages are being updated more frequently. People are using more of them, you know, they're using more and smaller packages.

And a lot of these, the, the way that malicious packages get in in the first place is by pretending to be something they're not. Right? They're all imposters.

Maybe they're using typo squatting. Uh, so that you think that something that is, uh, a capital I, you think that it's a lowercase l you think a hyphen should be an underscore, and so you download the wrong version. But it's not always about fooling humans.

Sometimes this is actually about fooling package managers, right? There's some of these that, uh, a human would actually detect quicker if, uh, a package manager thinks higher version number equals newer version equals better. Let me automatically update that for you.

42. You know, most people would look at that and say, no, those, that, those numbers don't seem like that's a serious real package, but the package manager is trying to be smart and ends up being dumb and bringing that malicious package on board. It's interesting.

And then just something as simple as typo squatting, uh, I heard one the other day, two ends next to each other, oftentimes look like an M. So as you're, ah, you know, yes, skimming by reading text, it's easy to gloss over those things and not realize that, Hmm, wait a minute. That, that, that looks kind of fishy.

So, Right. And everybody's in a, everybody's in a hurry. You know, this is why when, when we created our malicious package protection, we knew that part of what we needed to do with that was actually just block it from ever coming in.

If, if those packages are identified, you know, you didn't notice that two NS became an m well, this noticed it. This is, is reading it. This is detecting all of those malicious packages and it's enforcing that policy, so you just can't get that into your code base.

Mm-hmm. Talk a little bit about, now this isn't specific to the report, but I'd love to hear some more about mend. You know, it isn't a, we can scan our code once a month, we can scan our code every once in a while.

We've gotta fill out a compliance report. I mean, as much as we're creating, integrating continuously building, continuously deploying, delivering code, I mean, this has to be a part of the process of how we build it. Yes.

Not a secondary step. We do once in a while when we fill, like we might want to check it. Yeah, absolutely.

io, we, we keep telling people the best time to enforce policies, it's in the repository, and you enforce on push, basically, you know, you, you can detect, um, you know, are there critical vulnerabilities? Are there malicious packages, are there license types? You know, is somebody putting a G P L license in this that's going to cause problems for us later on down the line?

And you know, I I, I heard from somebody recently, he said, I can buy whatever tool I have the budget, I can buy whatever tools my devs need, but I can't force 'em to use it. Mm-hmm. And I said, that is actually, you know, that's where you're wrong.

And that's what companies, you know, really need to get involved in with best practices is enforcing this on push, making sure that there's kind of no way within your development processes to avoid that security step so that it's being done all the time. And just as part of your development, Well, anything, anytime you interrupt that cognitive, you know, focus of the developer, anybody else doing that kind of real deep thought work, they don't want to use something that's gonna get in the way, like every time I have to remember to do this or there's five extra steps or even two extra steps, versus when I do a push that happens automatically, it'll let me know if there's an issue that I wanna take care. Yeah.

And, and, and with, with mend, we, we figured that out about, you know, this, this kind of cognitive friction, um, as we call it. And, and we really made it so that you can do everything from the repository, whether that's, you know, looking at more detailed information, um, about what you've discovered, whether that's implementing a fix in your repository, you shouldn't ever have to leave that and go to a, a separate ui. You know, and, and part of kind of the deal you make with developers when you say, okay, you don't get to skip these steps anymore.

Okay, well, yeah, but you've gotta make it easy on the developers. You can't make it harder on them, or you're gonna get a rebellion. Yeah.

That won't last very long. They push back and hard. Um, I'm curious, so, so when folks don't have a have something like mend in place, what are the alternatives?

What do you do if you, if you don't have a mend today? Yeah, you know, I, I think there's a couple of, uh, there's a couple of, of what I think of as, as really painful and bad, bad ideas, uh, out there. So some people are just restricting their open source really, really hard.

I, I talked to somebody pretty recently where their company was doing this. They said, you know, we have one specific open source license type, and within that, we're going to allow this. If you want anything else to be approved, here's a forms based approvals process, right?

This is, this is a very irritating thing to have to go through. And I said, this sounds like it probably makes people upset and doesn't work very well, and it makes everything slow. He's like, well, yeah, we're never, we're, we're not hitting our deadlines.

You know, we're, we're not, we're not hitting what we promise. Um, we're, we're not releasing on time. And, and I said, well, you know, don't you wanna use more open source?

He says, yeah, but, but from our perspective, the security is too big of a problem. So that's one bad solution. Uh, another bad solution is to implement a solution where everything is kind of manual, where there is a lot of that context switching, you know?

Yeah. You're gonna have conscientious devs who just, they care about security as part of their job and, and they're gonna do it no matter what. Even if you make it hard, they're gonna do it.

But we know that is not all devs. Right. And, and it doesn't give you the coverage that you need.

Right. The devs who are most likely to be conscientious about that kind of thing, were also the most likely to be really conscientious in their coding practices in the first place. So, you know, they're gonna have the fewest problems in the first place.

You know, what you want is coverage over all of your devs. And so using kind of manual processes, using three tools and, and kind of haphazard ad hoc, it doesn't give you that kind of visibility, uh, over your, your organization from kind of a broad umbrella perspective. Interesting.

Uh, I'm, I'm curious, when your researchers were working on the support this report, kind of looking at the motivations, some of the reasons behind, now there's obviously the almighty dollar or whatever the currency is of your nation state, um, and that's certainly a big, big, uh, you know, incentive for folks. It isn't always the reason. What, what are some of the other other, uh, you know, incentives, why, why this might be introduced through packages?

Yeah, so, uh, we were actually really surprised, uh, to see that one of the, the biggest, uh, one of the fastest rising areas where malicious packages are taking off is, is in this area called protest wear. Um, so right now there's obviously, there's conflict in Russia and the Ukraine, uh, people have incredibly strong feelings about that conflict. And some of those people who have those really strong feelings are maintainers of open source packages.

Um, people are releasing packages that work exactly the way that they should. They, they contain all of the code that the original library did, except there's a little bit of malicious code in there that's buried deep. And what it says is, if this, if you find that you are in Russia, if you're deployed on a computer in Russian territory, start deleting files, start wreaking havoc.

And, you know, so for a lot of organizations, this package would actually function perfectly normally like a regular package. It's only when it's in that geographic region. And when you hear that, okay, so this is the start of it, right?

But apply this to geo geopolitical conflicts going forward and thinking about the, the fact that these are quadrupling year on year, uh, the, the amount of this kind of of protest wear, um, could be really, really significant as time goes on as well. It's kinda thing that can easily slip in and nobody notices until Yes. Right, Right.

Exactly. Because it's a, it's a good imposter. It has all of the code it's supposed to have, it just has that little extra payload that something special.

Mm-hmm. Interesting. So you talked about exfiltration of data.

Yeah. Things like that. Obviously stealing people's data.

There's also brand damage, you know, things like that are other motivations. Correct. Yeah.

Yeah. That, that can also, and, and, you know, a lot of times that comes down to money at the end of the day as well, because your customer's data is incredibly valuable. And, you know, th this is a place where I kind of foresee that there's gonna be lawsuits that happen because of the exfiltration that goes on, because sometimes these packages can be hiding for months or years in code and nobody knows that it's there and it's, it's just quietly doing this the entire time.

Interesting. Uh, well, let's talk a little bit about, um, I know spam was one of the issues when I was looking at the report was also, you know, I kind of, I usually think of just spam as things I have to worry about my inbox. Yeah.

Talk a little bit about that. Yeah, so, um, we noticed that there were a lot of spam packages, um, that basically existed to, to create spam to, to produce spam. Um, and there were also, uh, some of the one, some of what we identified were just, um, created by suspect publishers.

These are publishers who in the past have created malicious packages or have created spam packages. These are no longer trusted people. Um, and so even though these might not be able to wreak the same kind of havoc that an exfiltration or ransomware botnet sort of attack could launch that we still see these, these spam packages as, uh, as, as being a problem.

Anything that is from a non-trust source, we see that as being a problem and we identify all of those to our customers. Very good. Um, tell us a little bit more about some of the risk vectors.

Yeah. What are some of the differences like about attack and versus risk? Yeah, absolutely.

Um, so as far as risk vector trends go, um, we've kind of seen a few different, uh, risk vectors that, uh, we, we actually analyzed the malicious packages according to their risk type. Um, we looked at the frequency and the distribution of those, and we kind of created a, a profile, uh, that is first we, we, we kind of excluded all those ones that are, that are just spam. There were a whole bunch of those.

We excluded the ones that were just from suspect publishers. And then we kind of took a look at the distribution within what was left. We found some crypto miners, um, we found some protest wear.

Um, we found a lot of malicious obfuscated code and malware droppers. So these are, you know, intended to put malware into your code. Um, a lot of them were contacting external hosts.

Um, you know, there, there's actually multiple vectors that are being used in some of these attacks. And, and, and the most common was exfiltrating sensitive user system data. Um, that was 56% were, were doing that.

Uh, you know, they might also have the malware drop or they might also have some of the malicious obfuscated code. Interesting. Well, there's always such, you say great things that come out of reports, like not great that those are happening, but great that we know about it cause we can take action.

Again, it, if you had to, um, kind of put a bow on it and sort of summarize for what the takeaways were from the research, how would you describe that? Yeah, I think the takeaway is that you can't think of application security as just being about vulnerabilities. There is a bigger picture here, right?

This is, this is the supply chain. And the supply chain is becoming more complex and the more complex it becomes, the more opportunities there are for attackers. So, you know, looking beyond vulnerabilities into malicious packages and into just sort of thinking about AppSec as this broad more umbrella category where attackers aren't just thinking about specific vectors.

They're trying to look for the easiest way in and we're gonna have to stay a step ahead of them. Excellent. Well, we're, we're almost, uh, our a is almost upon us rss a c I presume you're gonna be there as well.

Yes, yes. Men do. IO is going to be there.

We will have, uh, a terrific booth. You'll be able to, to set up meetings and, and talk with us in a little private room right there. Excellent.

io site. Yes, I know. I've gone there myself and picked it up.

And, uh, we also have a show we're working on together, a series, video series on text Strong TV AppSec today. So go to Text Strong tv, definitely check that as well. Lots of good resources.

Lots of great, you know, that's the thing. We all need help, right? We all benefit from, uh, the, the work that we do.

And whether it's products, technologies, information, research, you know, it's all helpful. So appreciate you contributing to, to that body of knowledge. It's great to be here.

Thank you, alrightyy. Great to, uh, great to be talking with you. io.