Securing AI Models Against Cyberthreats with NCC Group’s David Brauchler

This is Textron tv. Hey guys, thanks for the throw. We're here with David Brocker, who's a principal security consultant for NCC group, and we're talking about the implications for cybersecurity when AI models are added to the IT landscape, as it were, because, well, they're a new animal in the zoo, and of course they attract a lot of attention.

David, welcome to show. Thank you very much. I'm glad to be here.

What should we be thinking about here? Because I think a lot of people are deploying AI models and they're very excited. We have a lot of irrational exuberance, but it's not clear to me.

They're thinking through the cybersecurity implications, which, uh, are many. Oh, yeah, no doubt. I mean, if we think about it, right, we've had buzzword after buzzword, more or less every year where an organization rushes to get into the next hot thing.

You know, we had IOT, we had cloud, we had blockchain, some of these pan out and become the next big thing. Others fizzle out over time. And for better or for worse, AI is that thing.

And so we have organizations who are rushing out to deploy either large language models or trying to implement, implement machine learning in other capacities within their organizations, but they're not properly understanding what the security implications of these systems are and how it impacts their threat model going forward. So what does that look like exactly? What are some of the immediate threats and maybe some of the ones that people don't appreciate enough?

Yeah, absolutely. So I think that on one hand, you know, the, the big buzz ever since Chad GPD came out was prompt injection. You know, people get this AI to say bad things or get them to like, more or less jailbreak the functionality.

And you know, on one hand it's interesting, it's cool, but I think that prompt injection is kind of like running JavaScript within your web browser. So I can pull up the developer console, make an alert box, show up, do whatever I want, but in the end, I'm just impacting myself here. There's not really further reaching implications, but when we have the opportunity to make cross user changes in either functionality, uh, in data states, well all of a sudden we have a really interesting implication to our attack model here.

So, for instance, um, if your ai, for example, has excessive agency, I might be able to exploit its ability to access resources that I as an attacker don't have access to. So let's say that I'm talking to a model and it can reach, uh, every account in say, a bank and I convinced that model to go and just start, draw withdrawing money from these accounts and sending 'em to mine instead. Well, all of a sudden, I as an attacker have access to information or resources that otherwise I wouldn't have access to.

Or let's say that a secondary user asks this system to compile a set of data and my data is contained within that larger set, I might be able to exploit the AI and tell it, Hey, while you're in the process of compiling this, you know, go ahead and, uh, send me all of this user's bank account information. And, and we've seen these attacks exploited in the wild already. Are we not paying enough attention to what is going into those prompts?

Because, um, we've seen people stick intellectual property in those prompts, and I think they're under the illusion that, uh, because the prompt won't be used to train the next generation of that AI model, theoretically, that nobody's looking at that data as they craft that prompt. But it looks like to me that that prompt is probably something cybersecurity folks are gonna be looking to steal. Yeah, I think that a lot of this isn't going to be some magical black box that we don't understand.

And, and that's really the current misconception that this is all brand new territory. But in the end, all of these components that we're creating fit nicely within our understanding of how threat models are laid out. So even if I can say, Hey, I don't necessarily understand how the AI is dealing with this piece of data, I can, I'll say, theorize it.

I I can, uh, theory craft it, if you will, as, okay, I'm placing the trust in the AI to do this thing. If that trust is greater than the trust I'm giving to the threat actor who can make the AI do things, then all of a sudden we have a problem. And so I think you're 100% correct.

We're not paying enough attention into how we're segmenting our intellectual property. And the big picture here is that we're running into the exact same issue that we've had for decades in the security space. We're not implementing proper data code segmentation.

And so we have these systems that are effectively consuming all of this data, treating it as code, and not properly putting trust boundaries in place such that I as an attacker, if I get my data to the model, can't influence it to do something malicious. You mentioned intellectual property, and I was just thinking about this not too long ago, but I know we're all obsessed with securing the access to the AI model, but it also seems to me that some criminals are just gonna steal the model altogether, right? I mean, it's, it's arguably some of the most valuable intellectual property any organization is gonna have.

Yeah, absolutely. And you know, we've seen several different iterations of this kind of attack, I'll say in the academic field. We've seen very advanced model extraction whereby repeated prompting we're able to more or less build a child version of the model that we're talking with.

Even popular, uh, open source models out there that you can see. Um, a lot of them have been trained on the output of things like GPT-4 and chat GPT. And so the model itself offers an interesting attack surface as a valuable piece of intellectual property.

And so if I have the capability to send unrestricted requests to that model, I might have the ability to begin drawing upon its knowledge base and training, I'll say a mini me in a sense and building it out, you know, a clone in a sense. Who's in charge of this? And I'm asking the question because, well, data scientists are hard to come by and cybersecurity folks are in short supply.

So who do we get that actually knows both? Yeah, you know, I think it's an open question right now. And when it comes to securing ai, there are a few different approaches that I've seen.

Uh, one being a bottom up approach where somebody starts from the perspective of let's learn everything there is to know about data science, everything there is to know about machine learning and build up so that we can later learn the security implications. Then you have what I would call the top down approach where someone starts in the security field and says, okay, let me understand how these systems are being deployed. As far as I'm concerned, it's basically a black box.

And from there we'll figure out what the security implications are for giving it a trust relationship to these assets. I think both approaches are valuable, and I think it's analogous to what we've seen in the security field with things like traditional computing, right? Uh, to do a say a pen test on a web application.

I don't necessarily need to know the ins and outs of how a processor implements an algorithmic logic unit, but having that information is still valuable in the long run. And so I think that security experts and machine learning data scientists need to come together to build out the best practices for the future going forward. I know that at our organization, NCC group, we already have a team of individuals who have been training from that application top downside as well as a select few individuals who have been training from the bottom up so that we can combine forces and best understand.

Okay, let's say that we're given a system that's implementing a language model, a classifier, et cetera. What are the very security attacks that we can pose against this in order to compromise valuable assets within the application? The attacks you've seen, are those attacks that requires the expertise of say, a nation state, or are they attacks that can be basically run by your run of the mills cyber criminal syndicate?

Yeah, the way I, I see it, I consider us in the wild west of AI right now due to the fact that everybody's implementation is different. Nothing is truly standardized, but I think that the current level of security vulnerabilities that we're seeing can definitely be exploited by a threat actor with sufficient knowledge of how these systems work. And when I say sufficient knowledge, I'm not talking about somebody who knows deep down the, uh, the cryptic secrets of how machine learning models process the, their inputs, but somebody who just understands an application architecture, how all these components have fit together, uh, can reasonably filter out different ways that they can chain exploits to get to these sensitive assets that they otherwise wouldn't control.

In fact, I put out a blog post analyzing AI application threat models and understanding, okay, if we start at these various places within the attack chain here, how we can op, uh, compromise this list of sensitive assets within our application architecture. So this is definitely something within the, the realm of typical cyber criminals. It's not something that you have to have years and years of experience to exploit.

It doesn't sound like there's a silver bullet for securing these models either. It sounds like we're gonna need something that feels more like a traditional layered defense. Is that right?

Exactly. I mean, defense in depth has been the go-to, and it will continue to be the go-to for the foreseeable future. You know, these right now, unfortunately, these models are set up in such a way that data and code are the same thing.

And unlike traditional attacks like SQL injection, we don't have an easy way to tell the ai, whatever you do, don't consider this piece of user input a a piece of code to be ran in a sense. Obviously there's some distinctions, but more or less that's the current state of play. But there are strategies that we can implement to segment that data away.

I could say create a, a, uh, model that's only responsible for data retrieval. So if I, for instance, get a prompt injection and it's collecting a set of Amazon reviews, it doesn't matter whether or not I convinced it to do something malicious because it's entire capability set it collecting reviews. So it works, it's going to have a, uh, say a negative review as its output, but then the model that's responsible for making changes to my account needs to be severely locked down and should never encounter malicious data.

If I have a model that's responsible for retrieving those reviews, it's output should be handled by traditional code not passed into another language model that could be influenced by malicious data. Will we need AI models to secure our AI models? Maybe.

Maybe. And, and we've seen a lot of success historically in using machine learning models for things like network intrusion detection for, uh, system intrusion. You know, if I have say, um, anti, well, we'll, we'll call it an antivirus located on my system, and it's looking for these indicators of compromise, we've seen that oftentimes those can do a decent job at detecting when an attacker has gotten on the system.

Obviously, like you mentioned before, it's not a silver bullet. Advanced threat actors can still get around it, and we have to have a baseline understanding of what a secure system looks like, which unfortunately for many organizations, they might not have the resources to to build that out. But in the end, there are advantages to using these systems in a defensive capacity as well as an offensive capacity.

Ultimately. What's your best advice to organizations about how to get started down this path? Do I just grab everybody, throw 'em in a room and lock the door and hope something good happens?

Or are there kind of baby steps I can take to get me down the right path? Sure. I think it starts with understanding what your organiz, like the current maturity of your organization.

You know, where are we as our, in terms of our security model, what have we already done to protect our critical assets? Because if you haven't addressed low hanging fruit, adding AI for defense purposes isn't going to help you in the end. There are things that attackers can do once they're in your system that aren't going to trigger any of any of these red bess, uh, or red flags, rather, you know, we've seen the same thing with web application firewalls.

They're a great strategy to make yourself a more tedious, uh, target for threat actors, but in the end, you're still just making yourself an annoyance. So the important thing is to address these low hanging fruits in terms of traditional security measures, and then go on to address these higher level defense in depth AI driven, AI powered, uh, advanced security infrastructure techniques. And so when it comes to securing the AI aspect itself, I think the important thing is to know what's going into your architectures.

What are we embedding into this platform? What are we giving it access to, and what does it have the ability to see? Because the second to threat actor has the capability to influence this model in ways that we didn't anticipate or host.

You know, we have the, we have given threat actors the opportunity to compromise assets that they otherwise wouldn't have had the ability to touch. So before you go out and begin implementing these in the wild, take careful note as to what you're putting into your systems and understand where things can go wrong in the process. Are we gonna have to wait?

I think you alluded this early, but let's be explicit about it. Do you think we're gonna have to wait for some sort of catastrophic event for everybody to wake up? Or can we, for the first time ever proactively go after this before we have a major issue?

Sure. Well, I think the distinction here is between can we and will we, uh, I I think we definitely can. And, and I think that the security industry in a sense has matured over the past couple of decades.

We're better at sniffing out security issues before they happen. Even take, you know, historical vulnerabilities such as Log four J, we did a pretty decent job once we found that that was an issue at patching a variety of systems that had log four J vulnerabilities. Obviously we're not gonna hit all of them, but that could have been catastrophic across the board, and ultimately it wasn't as bad as it could have been.

And I think that AI is similar in that sense. We're going to have security issues, we're going to have major hacks that are, that are caused by these, uh, machine learning models being implemented in context that we didn't think too, too much about beforehand. But in the end, we have the capacity to secure these models before they become an issue.

I am more, I, I'm a little more cynical about it. I don't know for sure if we're going to get there, but I hope we will. All right, folks, you heard it here.

I think that was a cry for help. It's similar to the issues we're run into with developers. They don't have a lot of cybersecurity expertise and they do things that drive everybody crazy.

Well, turns out that data scientists know even less about cybersecurity. So buckle up. 'cause it's gonna be a challenge.

Hey David, thanks for being on the show. Thank you. Enjoyed being here.

All right. And back to you guys in the studio.