Ross Bryant, Phylum | OSS North America 2023

This is Techstrong tv. Hello everybody. And we're back at the Open Source Summit and we're talking with Ross Bryant, who's a security reacher security researcher for Phum.

And we're talking about a new report that he's put together with his team about who exactly is launching these attacks against our software supply chains. What do they look like and what should people be looking for? Ross, welcome to the show.

Thank you. Thank you for having me. So give us the high points of the report that you guys put together.

I think a lot of folks are even dubious that anybody's attacking their software supply chains cuz they can't see it, so they don't know about it. That's the one. But what did you guys, So as we were looking through, uh, mostly NPM and ppi, we just saw an astonishing number of packages that, um, didn't seem to really do anything except just literally the ecosystem with this spam.

Um, the, some of it was SEO spam, some of it was, Hey, download this free e-book. Um, some of it was actually delivering malware. Uh, some of it was typo squatting and, and the variety of attacks that we're seeing is, um, too numerous to even like really articulate.

But the fact that it's there is pretty evident when you look at, um, the variety of, of, of packages that people are installing. So given that, should we just assume that everything has malware in it and, you know, there's some sort of malicious intent behind it, and how do we clean this all up? Yeah, I think that's hard because, um, you know, if I don't, if I don't trust anything, then I'm never gonna get any coating done.

I I'm, I'm definitely not gonna reinvent a nu pie on my own. I'm gonna trust it when I pip installed Nu Pie that it really is what it says it is. But, um, there's a lot of other packages out there that are, uh, or a lot of other actors out there that are trying to infect dependencies of dependencies, of dependencies of packages they're taking over from maintainers, uh, who just simply don't wanna maintain a particular package and then, and when they take it over, they put in their special sauce.

I don't think it's, it's the sky is falling just yet, but I do think people need to be aware that dear developer, you know, did you know that there are these pre-install hooks in your NPM package or your setup dot pie file that's curling some file on internet and installing it on your system as you have pip installing something. So illuminating those features or mis features to developers I think is key to even beginning to wrap our brain around the scope of the problem. Do you think we appreciate how sophisticated the attackers have become because they are just basically putting together a bunch of components and then hoping that somewhere the malware gets turned on some day and we don't really think that through as well?

That's right. I, we see sophistication all up and down the spectrum. We've seen some really bespoke, uh, obfuscation schemes, uh, in many, many, many layers where the malware's finally delivered at a deep layer of obfuscation.

And then we see people who are just literally flooding the market with packages through automation and they don't have to be sophisticated. They're almost like the, the email span problem of not too long ago where I just, if I make a million things and I get 10 people to click, it's already a value proposition for me. The barrier to entry for these attackers is so low and their reward is so disparate that a lot of 'em don't have to be good to be effective.

So they're getting highly automated and it looks like they're investing their ill-gotten gains into more sophisticated attacks. Sure. Um, are we correspondingly making the right investments to defend against that?

Or is there an imbalance in the, in the forest as they say? Oh, that's a great question. You know, as long as we're in the open source free software system where anybody can upload anything, I think there's always gonna be this danger of anybody can upload anything.

And so now we get down to, well, how do I trust this package versus that package? Um, it it, it's a hard problem. Do you think maybe we'll be applying AI to this in the future to kind of analyze some of these components to source that out?

I, I, I think we have to simply because what we see in these obfuscation schemes is not written by humans. It's code that was given to a black box to write code that is obfuscated. Um, and, and so you have to have computers to compete against computers and code to compete against code.

Um, I don't see any way that, uh, manually triaging all the packages in a given ecosystem is viable. If it ever was. What should the average developer be looking for?

I mean, should I just pay more attention to the Euro URL of the repository cuz somebody's doing some typo squatting thing where what can we do to kind of help ourselves? Yeah, that, that's part of, uh, the problem we're trying to solve. Um, there are many innocuous things that if you're a, uh, a developer in a certain community, when you see something in a, in a file is that, that looks very suspicious, that looks very strange to me.

Whereas somebody else who's maybe not as familiar, maybe new and learning the system might just download and and, and move on. I think that part of the problem that it for the attackers is that it is very, very difficult to mimic perfectly. Every single feature of a legitimate package is very, very hard to look exactly like, um, something that we would trust and download something probably represented by one of the people in this room.

And so as long as there are these little mis features that attackers, uh, can remain lazy about not fixing that leaves defenders and developers the ability to go, wait a minute, that looks strange. What, what is that doing here? And maybe give them a clue that something is suspicious, something is up, something is just not quite right with this package.

So do we need a massive, if you see something, say something campaign, I, I think certainly the community has to rally around this problem. Um, because, uh, what is it? If there's enough eyes, all bugs are shallow?

Well maybe if there's enough eyes, all malware is shallow. Um, no one is a polyglot in all the languages that are represented open source. And so yes, I do think communities need to kind of circle their wagons around their developers and say, what can we be doing to identify and, and detect and, and alert when like, Hey guys, this package that was published two hours ago looks really bad and you should know about it and avoid it if you can.

Are we too cognizant of maybe trying to shame people so they don't come forward and kind of cough it up and say, yeah, I made this mistake, so maybe we should have like, you know, a general amnesty program for Everybody. I think it, I, I do think there's room for us to be a little more human to one another in this regard. Um, you know, typo squatting is a really popular, uh, attack type and if you just a couple months ago type Matt PT lib in your PIP install instead of Matt plot lib, you got a crypto wallet, uh, clipboard attack.

Um, we shouldn't be shaming people for that. We're all human, we have human frailties. Um, nobody's gonna catch everything.

And so yeah, I think there's room for us to grow as a community to really rally around like, this is an attack on human frailty, not an attack on a system or an abstract entity that isn't a real person. There are, however governments around the world put with imp pending legislation that they're looking at that's gonna hold developers more accountable. Is that a good thing, a bad thing?

Is it feasible? Uh, that's a good question. I guess it depends on the, uh, how it's actually implemented and how it's enforced.

Um, I I think you, if you have too heavy of a hand, all of the innovation for all of these things that we take for granted today is just going to be driven to zero. Um, I I I, I'm not gonna say it can't work, but I'm just gonna say it's not gonna be very compatible with an open source ecosystem as we know it today. Just put this report out.

So of course your boss probably said the following, what's next? What's Next is, uh, we're, we're continuing to track these attacks and, and really trying to see where attacks have sort of ebbed and flowed, right? We, we'll see something surfaced for a month or two and then it'll go dormant and then there'll be some variation and then it'll resurface again that the, the, the external wrapping has changed.

But when you pull it back the layers like, wow, that's the thing we saw a few months ago. And so really watching how the attackers are adapting and developing, um, I have no doubt that as we get better at detecting things and reporting them, they're going to adapt and change their tactics. And so it's just gonna continue to be this arms race.

It Also seems to me like once they get a tactic that works for one language, they go see if it works across multiple languages. Absolutely. And that's why you see these different waves.

Absolutely. Yeah. And that's one of the problems that we have, again, going back to the community that um, no one person knows all the languages.

And so if an attack is really popular in, in a Python ecosystem, maybe somebody's gonna try to do that in Erling or you know, Java or something like that. There obviously certain, um, ecosystems do have some guardrails. I'm not saying it's completely the wild west that it's harder to pull off something in one language that is in another, but the attackers are really, um, they're really determined to make our lives miserable as best they can and be profitable with it.

Are these attacks from maybe organized crime folks or are they nation states? Or are you starting to see specialization in the bad guys as they kind of build out their own development tv? Yeah, I, I think it runs the full gamut, right?

From all the way from nation state down to just what we would call like petty vandalism, just people doing stuff just cuz they can, um, attribution's hard even when you have a rich set of resources to try to figure out who is doing what. But yes, I think I've seen indications that, uh, this seems to be more of an organized effort. Maybe this might be a less organized effort.

Um, but it, it, you name it, it's out there. Who's supposed to be responsible for this ultimately? Is it the developers?

Is it gonna be the DevOps team or a bunch of cybersecurity folks? You know, I would like to say it's everybody, but then that means it's nobody's, um, I think people just have to decide for themselves, like, okay, what are we really going to do to look at, get some look inside? What are the packages that we're installing into our system and has anybody even taken a look at them?

Has anybody ever even bothered to go through and see what's in here? Um, above and beyond the things we're seeing about, you know, SBOs and, and all of those other sort of external structures that at the end of the day, the attackers are, uh, motivated to evade all of those defenses as best they can. You've been doing this for a little while now.

What's that one thing you see folks keep doing and this makes you shake your head and go, I think we're better than this Boy. Um, we've seen some campaigns where, um, unlike a typo squat where I'm just, you know, I, I mistype two letters. Uh, there's been a couple campaigns where people are just smashing words together and publishing two or 3000 packages where it really makes you wonder like, who do you, who do you think is going to install this?

And there are a couple of good answers we have to that, um, in that maybe you don't directly install that, but if I can inject that as a dependency way down in your dependency tree and you don't know that it's there, then you get this bespoke piece of malware. But yeah, there are some real head scratchers out there that we've seen. We're like, what do you think you're trying to accomplish?

And uh, ultimately we're just left to guess and speculate. But, um, yeah, that, I don't know. They, I don't know how smart they think the developers really are or aren't.

Uh, sometimes when it comes to installing these packages. I think part of the issue of course is education and is is there some way to go after that? A lot of developers, I know cybersecurity was an elective and if it was an elective, they didn't take it.

So how do we go back in and kind of say to these folks, Hey, um, we need you to kind of get retrained on this stuff. Yeah, Just simply being aware of what's out there, right? I mean, if, if, if I don't think that there's any problem with me pip installing or MPM installing a particular package, whenever I feel like it, I'm probably not going to care.

And so as we see more of these attacks and write on them and, and publish them, we try to, we try to educate as well as informed to say, look, this, this is probably something you didn't think about, but it's out there. Um, so I that that, that's a really great question and I don't know that it's gonna be a formal education more than just a, a general awareness that dangers out there. Will we see more repositories that are vetted and just say, uh, only download code from this particular spot and you know, and we'll have some authentication to make sure that that thing is really what it is.

I think so. But then again you run up against sort of, you know, what's a fundamental principle of opening free softwares that we're all out here in one big ecosystem and if we begin to wall off these places, places, uh, to where doubt shall only install software from this repository, well now I'm just moving that trust from, you know, one point to another and uh, you know, there could be some problems there but um, I think it would also stifle innovation again, you know what if I'm a a brand new coder and I've got some whizbang idea and I wanna put my idea out there and I'm not in this walled garden repository and how are people gonna hear my ideas if I'm not in there? So that's a tough one.

Are we in danger of seeing the benefits of not just open source but the internet as a whole start to kind of bifurcate a little bit more? Cuz we're gonna see people kind of split up into their own little geopolitical universities. Of course.

Yeah. And you know, we won't have that level of collaboration where That's exactly right. And, and, and who knows, that may actually be a motivating factor for some of these attackers is to get us to fragment like that.

And so then at that point we just have to decide as a community, like, okay, are we going to do that or are we going to really say no, we're gonna maintain our open and free software, but we're gonna do it in a way that ensures that, uh, mistakes made by developers through human frailty or through just, you know, lack of knowledge aren't going to hurt them as badly. Um, yeah, we're going to have to decide as a community what's the best thing to do there. All right folks, we got a big question in front of us.

It is, what are we gonna do as a community that said, we all gotta get better at writing software that's more secure, even when it's free. You gotta look twice cuz free can get expensive. Real quick.

Ross, thanks for being on the show. Thanks for having to me And we'll be back in a minute.