ChatGPT’s OSS Malware Review – Henrik Plate, Endor Labs

This is texturing TV. Hey guys. Thanks for the throw.

We're here with Enrique who is a security researcher with end or labs and we're talking about chat GPT and who isn't these days Henry welcome the show. Thank you so much for having me. You guys just did a study looking at the kinds and types of code and the quality of the code and the security of the code that can be generated from these platforms.

Marcus through what you found because I think people might be a little surprised. Yeah, maybe maybe a quick summary at the beginning is the main takeaway from the study. We have conducted over a couple of what I mean the setup took a couple of weeks, but the main result of the actual experiments conducted was that despite all the progress that is made both in terms of cold comprehension that is understanding and letting GPT and other large language models explain code as well as code generating.

Those models are not yet ready to really replace humans when it comes to classifying. Open source packages as malicious or as benign which is as you know, probably a Hot Topic as well these days there are so many hundreds and thousands of both in Source packages published on a regular basis many of which or an increasing number of which contain malicious code Source in short supply chain attacks. So good news for manual malware analysts.

Check GPT and other llms are not yet there and not yet ready to replace them. It doesn't give me a head start down the path and kind of help me analyze more stuff faster than if I was just doing it myself. Exactly.

That is how we use it right now. So we seeing that it cannot really. Kind of in 50% of the cases where czechi PT concluded that an open source package is malicious.

It was actually what we call a false positive meaning it wrongly assess it as malicious while it was in fact be nine right and due to these deficiencies and the current let's say lack of capabilities doing a proper classification. We right now uses as an additional signal as an additional rev viewer with the human having the last word as to whether something is malicious or benign. It sounds like Almost the point where well, we're going to check the homework right of the machine and if we're gonna check the homework in the machine, they only hosts so much value because you got to review the code anyway, so what exactly is the benefit?

No, that is I think a very perfect summary of what uses of large language models should do these days. You should use it such that you can still very fire the output produced by the large language model, right if you let's say a good example is what we have done in this experiment. You have a snippet of code that you have found in an open source package that is potentially malicious, for example, because it shows the same characteristics and behavior as other malware, right maybe downloads something and then it is executing it on your local machine or on the developers machine right away.

Um, And so the assessment as to whether the snippet is is good or bad is something that you can with a review and verify manually without any problems or post to other tasks. If you ask llms check GPT to summarize a book, you will receive a very well written summary. But unless you have read the book you cannot say whether this is in any way accurate or not.

And so this is my my one of the Lessons Learned is use it or tasks that you can quickly verify such as malware assessments. Do we need a large language model specifically tuned for this function in this task versus trying to use a general purpose large language model. In indeed that is also a very good question.

Thank you for for that. I think that the way we have used it in the experiment as you say that was the general purpose modern but going forward. I think we will see find tunings.

On maybe specific packages or repositories source code repositories in order to refine the pre-trained model and does get better a better results. Maybe you probably know that GPT has been trained and so much. Um information taken from the internet of lot of which is probably not relevant for the task that we try to solve here.

So indeed, I think going forward we will see smaller models. Better trained more specifically and also refined specifically for the task at hand. And the same issue would apply with the recommendations for the code Snippets.

It may come up with the fix a particular issue that code is not likely to be the best code and it's gonna need to be reviewed heck that code may not work at all. And you mean when it comes to generating code, yeah. Yeah, and I mean this is this is a little bit of and different from the experiment that we run right so we did it used it for what we call code comprehension letting people let it explain software to people.

Um, but you're right when it comes to code generation check GPT. They they kind of make the same programming mistakes that are done by humans. So you will have these out of bound array accesses you have division by zero problems being introduced.

So again, if you generate it for code if you use it for generating code, We use General you should make sure that you can verify it right and that you that you actually verify it. So you should not just blindly copy and paste open that got it, right. Of course AI will continue to advance and so do you think we'll ever get past this current Log Jam we have where the people who build open source software.

They don't get paid. It's a hobby for them. And when a vulnerability shows up, you know, they're not gonna rush to go fix this stuff over the weekend for somebody because well, it's not their job and what will the machines kind of at some point narrow that divide or kind of help us bridge that or is it just gonna take some cold Hard Cash to make it worth their while There are indeed applications and prototypes that.

Try to automatically repair buggy code and indeed they and those those prototypes automatically create fixes and pull requests that open source maintainers could verify again and then merge they find it is a good quality. And so there is indeed a huge potential also to for code refactoring or how to repair and that can relieve open source maintainers take a little bit of their burden. But again, With the current quality these verify that menu.

Do you think people are gonna be tougher on open source code and and create all these dependencies in our code as we go forward or is it going to be just you know businesses usual and we're suck up their risk as we go up. No, I mean we bend a lot of time with open source. We follow the regulations.

We follow also the discussions taking place in the open ssf and other open source foundations that are so very well aware of those problems and that work on Solutions and overall. My impression is that we are not getting tougher on open source maintainers we the industry Acknowledges that those guys are spare often Spare Time developers and that it is the responsibility of the industry the responsibility of the consumers to to care and support for the for the stuff that they consume and build into their applications. And so that is a very positive development because indeed sometimes you could have you could get the impression that they are these kind of kind of accusations made towards open source maintainers that I never liked and which are not justified.

So what would be the role of the people consuming open source software? Do they need to contribute more people to help them maintainers or do they need to just contribute some money back to the community or do they need to just help with small stuff like documentation? I think any any support is appreciated.

But I think what companies consumers in general need to do is to develop a holistic approach to open source dependency management kind of have metrics that let them understand and measure the risk. That is coming with a consumption of Open Source. And then once you have those metrics take risk based it's a decisions as to do I consume it do I pull it into my application or if I pulled it in the past do I keep it or do I better throw it out?

So I think there's this. Um, yeah risk based dependency management that I that I find very important and which has to be kind of adopted by open source users more broadly. I think what we're saying here is user beware.

If you don't know where the software you're using came from chances are you've got an issue and you might want to think it through. definitely it is whether kind of but now people all the developers should have understood that they cannot blindly consume open source components any longer you should have visibility on the whole dependency Tree on the body graph of all the the packages that you automatically pull into your development project so that I hope is understood by more and more Developers. Do you think politicians get this?

It seems like there's a lot of legislation floating around where we're going to try to hold certain maintainers accountable for Flaws and software and it seems like kind of productive. Maybe I don't know I am this is yet another good news. I completely think that the politicians and The Regulators get it.

So if you follow all the you know, the regulations and all the initiatives that have been started and after President Biden's executive order back in 2021, the there are so many reference to the secure consumption of Open Source, including you know, the the software business of materials. And so I think this topic is covered and is understood. Once you're ultimate best advice then and who's in charge of making sure the software is secure.

Is it the security team or is it the devops team or is there somebody in between now needs to show? Oh, that is I think that is a that is very specific to the different software development organizations, depending on their size depending on their organization. It's set up different organizations.

Yeah, we'll just do it differently. So I don't think there is a one size fits all approach as to who Is the one taking care of it what is important is that the organization as a whole takes the responsibility for all the code that they consume no matter whether it is cold. They have developed themselves or whether it's called that they pull from open source projects.

Then who in within the organization does what? Can be very different. All right.

Well, you heard it here folks. Everybody needs to do something. It will maybe slightly different.

But the one thing for sure is if you do nothing it will end badly Henry. Thanks for being in the show. It was a pleasure again.

Thank you so much. back to you guys understand