Software Supply Chain Cyberattacks with Phylum’s Ross Bryant

This is Textron tv. Hey guys, thanks for the throw. We're here with Ross Bryant, who is chief of security research for phylum, and we're talking about how to secure our software supply chains because, well, it turns out that there's been a spike in the attacks launched in their direction.

Hey, Ross, welcome to show. Hey, thanks for having me. You guys have this report, but walk us through the high points, but I guess my first question is, is, is this a big jump in this spike or is this kind of a big spike off of small number?

I mean, how many software supply chain attacks are there these days? So we're seeing attacks numbering in the tens of thousands when it comes to, um, all the ecosystems that we look at. We're looking at seven different ecosystems, NPM Pi, PI, cargo, Ruby Gyms, Maven, NGE, and Golan.

And, uh, we're seeing steady progress by attackers in each of those to continue to attack developers who just simply aren't looking for, uh, malicious dependencies in their software. What is the nature of these attacks? Are they people stealing their credentials and then dumping malware in there?

Or is it more malicious or more sophisticated? What exactly am I supposed to be on the lookout for? So we see all kinds of, uh, levels of sophistication, obviously.

Um, simple attacks of just like children stealing discord tokens from one another. Uh, we see that all the time. It's, it's pretty prevalent, but there are a number, a significant number of sophisticated attackers in these ecosystems, um, that are doing a lot more than just simply trying to, um, give, give you ransomware or, or steal, uh, some tokens.

They're actually trying to steal developer credentials. Um, or, or worse, put remote access tools on developer machines so that they can then pivot out and move laterally through organizations. What should we be doing to protect ourselves that we're not, I feel like we've been talking about SecOps for a while and whatever, for whatever reason, it seems like it's a struggle.

That's right. Um, it still shocks me when I go to conferences and talk with developers, CISO, security engineers, who just don't simply think of this as an attack vector that people just, uh, don't think of open source malware, um, as a thing. Uh, you know, you still think of don't click on this attachment or don't browse to that, you know, uh, link people are just blind to the fact that there's people out there attacking open source software, uh, supply chains.

And, um, I mean, just simply being aware of, it's the first thing. Second of all, you need to actually be looking at the source code that you're pulling into your environment. You're pulling into your, um, enterprise.

Um, a lot of times we'll see things that the package itself that is published isn't malicious, but it has a dependency, which is, and uh, chances are, I don't think many software developers actually manually review not only the packages, but every dependency in that package for maliciousness. And so there needs to be some kind of automated system that, um, inspect source code, and that's what we at Lum have been trying to do. Is there some sort of blind faith in that the maintainers have somehow or other taking care of this whole issue?

And the assumption is that everything in that package is, is good to go. And maybe that's just part of the issue is we're blind to it 'cause we think somebody else did it. That's right.

That's right. I, uh, what is it if all bugs are shallow, if there are enough eyeballs, um, but software developers have a day job, right? Their, their job isn't to be, it is not that they're not security conscious, but they have code developed, they have features they want to push, they want to make the latest, greatest thing available to their users.

And that just simply requires, Hey, I'm not going to build a library, for example, for Unicode support. I'm just gonna go download that one. Oh, sure.

It looks pretty good. I guess it's got a few downloads. Um, one of the things that we've seen recently some attackers are doing is that they will, um, inflate their download numbers clearly through some automated process.

So it's like, oh, well that package has 250,000 downloads. It must be good. Right?

And really, it only has a few downloads and the other 250,000 are just some script out there pinging and, uh, automatically inflating their downloads. And so, um, yeah, it's, it's, it's, it's socially engineering the developers, uh, and the reason it's so successful is they just aren't looking. They just aren't looking for these attacks.

Not to just focus on the maintainers though, but it seems to me that developers are downloading components using them and then never upgrading them and not realizing that vulnerabilities have been discovered or there's been new updates to address a particular issue. And, um, so that just sits there in our production environments waiting to be exploited. So how do we make this whole upgrade cycle less painful?

Yeah, that's, that's a great question. Um, you know, I I, I don't think it's ever going to truly go away. It's just simply always going to be an arms race between attackers and their victims.

Um, one of the tactics that we've seen recently, uh, that we note in our report is, um, be it hours, days, maybe even months. Sophisticated attackers will publish packages and just leave them dormant. Uh, they'll have some functionality that does something.

Uh, and then after a few, uh, upgrades, you know, version one, version two, version three in version eight, they'll simply, simply stick some malware in there. And, and by that point, the developers become so comfortable that, oh, well that package is doing what it's supposed to be doing. Um, the developers are, are, are playing a long, uh, the, the attackers are playing a long game here.

And it isn't just simply upgrade everything as soon as you can because that upgrade might actually have the malware in it that the previous version didn't. Do you think there might be some way that AI might save us from ourselves here, because we'll be able to find these vulnerabilities and maybe understand what it is we're upgrading and why? Yeah.

I, I, I think it's going to get better. Uh, I haven't seen anything yet, although I, I keep watching for, uh, significant improvements in AI in this particular realm. Um, finding vulnerabilities I think is actually, uh, pretty straightforward.

There's in, in, in the sense that a lot of people are doing it and doing it well, but finding novel attacks, finding attack vectors, um, uh, that an attacker has just created, it's almost like a, something analogous to a zero day, uh, you know, in, in, in source code, uh, will find a little piece of opaque code a a a long encrypted or obfuscated string. And when the code is installed, the script is called and that string is unwound and it has a call out and it goes out and fetches whatever resource on the internet, uh, it's looking for. And that's what drops the malware on your machine.

So it's, it's, yes, we need to be vigilant about vulnerabilities, but there's so much more to these attacks than just simply, I wrote a code that, you know, has some known CVE against it, And we need to figure out what the relationship is gonna be between the cybersecurity folks and the developers. 'cause I feel like so much of this is still, I found a bunch of vulnerabilities. I stuck them in a spreadsheet, I tossed them over the other side, developer looked at it and said, Hey, we're not using that code B, that application is not internet facing, and then we ignore the rest.

That's so How do we kind, you know, make this a practical working relationship? That's right. And that's something we really, uh, we really work, work on a lot.

We, we ourselves are software developers and the friction between security and DevOps velocity is real. Um, you know, at vi we, we we're very sensitive to not we breaking builds unless we are absolutely positive that, hey, that package has malware in it, we're, we know that is bad. Um, because any tool that isn't tightly integrated with a developer's normal cycle of development, um, they're simply either going to avoid, ignore, or just not use.

Um, and so we have to be careful not to, uh, raise too many alerts, raise spurious alerts, and really be on the developer side of them being productive and security conscious at the same time. So what we'll often do is just simply raise informational things. Hey, dear developer, did you know that this code has obfuscation in it?

Or it has a call out to a strange URL and it's going to download that. Write a file to your system upon install or upon upgrade. Um, you might wanna know that ahead of time before you go ahead and install this package.

It's up to you if you would like to, uh, but we are at least warning you we're, we're at least trying to give you a set of eyes that you didn't have that review the source code that you're importing and telling you things that are usually indicative of some kind of strangest, if not maliciousness. I think developers are usually grateful for the advice, but part of them is also saying, you know, this whole shift left things feels a lot more like shift downhill than we also know what else rolls downhill. So yeah.

Is the cognitive load too high for them and do we need to kinda streamline this thing? Yeah, yeah. No, it, it is absolutely true that, um, they, they, they're not going to, they can't quit doing their day job.

They can't quit writing code. They can't quit, cannot quit shipping features. Um, and so we have to be on their side to say, Hey, we're, we're looking over your shoulder so you don't have to look at these packages.

We've at least already, uh, gone over these at least once. Mm-Hmm. Ultimately, what is your best advice to folks about how to get started down this path in a way that, um, is productive?

Do I lock everybody in a room who's has a vested interest until reason prevails? Or is this kinda like, I just gotta go walk around one developer at a time and buy a pizza and walk 'em through it? I I'm much more in favor of the latter.

I, I think the top down thou shalt approach, uh, you know, from, you know, heavy handed security people just in my experience just doesn't work. It, it, it doesn't get you, um, the type of productivity you need out of your developers. And so there has to be a, uh, you know, a common ground that, hey, we all have a vested interest in in our organization and our enterprise and our mission to have safe and secure code, but let's do it in a way that we still get the feature shipped to our users that we want in a timely fashion.

Um, but you're right there, there you, you've identified not a technological problem, but a sociological problem where, uh, you've gotta get people in a room to agree that, you know, security's not the enemy of product productivity and neither is productivity, the enemy of security necessarily. So we are at the end of the year, and of course, that always begs predictions for the coming year. Is this gonna get worse or better?

No, I I think it's going to continue on. Um, you know, one of the things that we've seen an uptick in that was most notable is, uh, we found some very sophisticated software in June, um, more sophisticated than we had seen before. And as it turned out in July, uh, GitHub published a post that said, um, north Korean state sponsored actors were actually behind the malware that we found.

And, uh, when we found it, we didn't have that data. They did. Uh, but we've since joined in a, in a partnership with them because what they're seeing and what we are seeing is that it isn't just the script kitties and the ransomware guys and the, uh, crime people.

It's it's nation state actors, and these are people with serious resources, serious sophistication, and they're getting into this environment. And so I, I fully see that, uh, for foresee that this is only going to get, um, worse in the sense that, um, as long as people aren't paying attention and it's a fruitful and cheap, and the barrier to entry is so low for attackers in open source software, um, they're gonna continue to attack developers, I feel like so much of our focus is on essentially trying to draw a line in the sand and say, from here on out, we'll be good. But last time I checked, maybe only less than 5% of known vulnerabilities are actually exploited, and the other 95% are still waiting to be exploited.

Um, so do we need to kind of go back in and look at some serious technical debt here on the security side of this equation? And maybe our SecOps processes needs to be on upgrading the existing applications as much, if not more so than the new ones? Absolutely.

I mean, if, if, if your company doesn't have any auditing whatsoever, or the auditing is really just, uh, a box check checking exercise, um, I don't think you're really taking this problem as seriously as you ought to. Um, but you're right, that, uh, what, what auditing firm is going to go vet, you know, the 10,000 dependencies that your application has, um, by hand. And, and, and that's exactly what file is trying to do.

We're, we're not doing this by hand. We have an automated platform that when you point your project at it, it will scan, go through and say, here are all the things that we found Yes, about vulnerabilities, yes, about license risk, but more importantly, indicators of maliciousness. Um, we have one customer who, uh, called us up one day recently to thank us that they had, uh, mistyped the name of a file that they were updating, and lo and behold, it was full of Chinese malware.

And they said, you, you guys saved our jobs. Uh, thank you for that. We, we would never have found it otherwise.

It's the, the, these complex systems are so complex, you, you can't possibly hope to debug them manually. So, uh, having some sort of automated system has to be part of the solution. Do we need to slow down the pace at which we we're developing applications to address security issues, or can we have our cake and eat it too?

Uh, I don't see that being viable. I don't see any, uh, engineering managers saying, okay, we're gonna, we're gonna ship less these days, uh, because, um, you know, the competi their competition is real. And, and so, you know, you, you have to fix the plane while you're flying it.

Um, and in that sense, I, I think that we have a pretty good solution to this. Um, just integrate this with your normal CI/CD build pipeline. Uh, go through one of our integrations that we have and, and it should be seamless.

It should be effortless to just work this into your normal workflow. And then you'll find as you go, oh, that package, that dependency, that has a strange update. Let's, let's go take a look at that.

Because once I, I, I, I don't doubt that plenty of companies have, you know, great code review practices for the code they wrote, but what about all the developers that work for them that wrote the open source software, right? They've never vetted their code, they've never been to their code review. And so if we at filum can, can illuminate these packages and say, you might wanna take a look at this, this is strange.

Um, I think you'll, you'll, you'll see it at least a, a, a a, a better, uh, feeling that people get that, um, you know, Hey, we, our code's good. We, we, we, we, we've looked at our vulnerabilities, we've looked at these holes and, and we're taking care of them. All right, folks.

Well, you heard it here. We are essentially enabling these people. They are dropping malware into open source components waiting for you to download them.

Then they'll sit back and when they malware, phones home, they'll start laughing all the way to the bank at our expense. Maybe we should put a stop to that. Hey Ross, thanks for being on the show.

Thanks, Mike. And back to you guys in the.