Cybersecurity Challenges in Operationalizing AI with Fortanix’s Richard Searle

This is Textron tv. Hey guys, thanks for the throw. We're here with Richard Searle, who's Chief AI officer for tix, and we're gonna be talking about all those things related to security there.

We're not fully thinking through when it comes to ai. Richard, welcome to show. Thanks, Michael.

It's great to be here. Thanks for having me on. I think some people are kind of loosely aware of the idea that maybe I shouldn't expose sensitive data to an outside external entity that I have no control over, but who knows, other people, maybe not so much.

But why don't you walk us through some of the things that we should be thinking about when it comes to AI security that we're probably overlooking. Yeah, for sure. Well, this is a really complicated area.

I mean, uh, there's, there's a lot to go at here. Um, AI has risen in the public consciousness over the last two years. Uh, a lot of companies are experimenting with it.

Some of them are looking to put it into production for various reasons. But what's happened is, uh, typically with innovation, concerns around security and privacy have largely been put in a second order behind the, the development of the technology itself. And we've seen that that's created risks for companies in the early days.

There were specific examples, um, for example, for Samsung, for, for one, where, uh, individuals were using services. They were communicating, uh, confidential information to those services via the prompts that they were using. And then that, that information was ingested by the model, uh, used in training, and then, uh, it was recovered by, uh, unauthorized party.

So I think you know that for chief information security officers, uh, chief data officers, there's a lot of concern around how this technology is being implemented and what kind of safeguards they need. And there's a number of issues, particularly with generative AI today. Uh, not all AI is generative AI, obviously, but, uh, that's the area of focus in most of the, the organizations that we're dealing with today where, uh, an understanding of how those systems operate and some of the, uh, potential pitfalls, uh, that are, uh, inbuilt within them, uh, needs to be, you know, more broadly understood, and then people can think about some of the safeguards they need to put in place and that those are the kinds of things that we're helping our customers with.

Well, let's dive into the different issues that we may find, and let's start with end users and sensitive data. Um, theoretically, I guess, uh, there's a box somewhere that I can check on my user agreement that says that the data I'm sharing with the LLM will not be used for training. Um, yeah, it seems to me it's kind of a hard to find that box, but b um, can I trust that or will the terms and conditions subject to change?

Well, I think one of the problems, I mean, certainly we've seen, um, public API services implement, um, those kinds of checkbox uh, um, protections for, for users based on, uh, you know, attacks like the one I I I mentioned previously. As you, as you rightly say though, what you're doing is you're transferring information into those systems and you are dealing with an untrusted, uh, resource in the same way as you are with, you know, things like cloud infrastructure, for example. And so, you know, you have to question whether you want to take that service provider's word for it.

But that's just one instance of how these services are being implemented. A lot of customers are as well, looking at things like local model deployment using open source technology. And so that's a whole different ball game because there you are actually taking the onus and the responsibility for protection of data being used by those systems.

And that comes down to, uh, what kind of information should be actually communicated to those systems, what kind of tasks, uh, they should be applied to, and then how you monitor that, those safeguards are actually effective. Uh, and so there's a, a balance to be struck between, you know, um, uh, offsetting your, or, or rather, you know, putting your security and your data privacy in the hands of a third party provider or, um, how you manage that internally. And of course, what we're also seeing now is, uh, hybrid systems to, uh, manage the cost effectiveness of things like generative ai, where you have a blend of both local and, um, you know, public APIs.

Uh, and so actually making sure that they operate, uh, with sufficient safeguards and, and information exchanges governed properly is a really important concern. And, and it's something that has to be addressed by companies in order for adoption of ai, uh, and the benefits that we expect to come from it to be, to be mainstream. And then to your point about folks who are either relying on a, a third party or customizing one themselves and kind of putting that in their own local data, um, it also seems the output of these models, regardless of who creates them, is subject to being poisoned, perhaps.

And I will get the, the wrong output, or, and it won't be necessarily a hallucination as much as it will be a deliberate attempt to manipulate an outcome in a way that gives me something that, whether it's a piece of code or some sort of information that I'm relying on that winds up being patently false. So how prevalent is that? Yeah, that's a significant problem.

And, uh, you know, there's a number of different, uh, attack factors that, that people can use to, to compromise these types of systems. You mentioned one in terms of things like, uh, data poisoning and training. You know, that's a, a well established technique that that applies equally degenerative AI as it does to, um, you know, more, not more standard reinforcement learning processes.

But there's also issues where you're using things like open source code where, uh, we know for a fact that there have been, um, uh, cyber attacks perpetrated, uh, or attempted to be perpetrated using open source code where, uh, to intents and purposes, the system might function perfectly well. But there might be things like trigger points in there, there might be compromise of the model parameters. And so what that risks is the potential for under certain circumstances, the model to misbehave or to compromise the process that you are depending, uh, upon it for.

And I think one of the real challenges for, for people moving forward is thinking about, you know, the business criticality that's gonna be built around these models and what the risks are, um, where information is either leaked by the model under things like a jailbreak attack, where, uh, it's possible to recover information through, um, careful prompting and manipulation of the model behaviors or, um, the reliance upon code, which is, uh, effectively untrustworthy. And that, that's one of the things that we are looking at very closely at the moment. And I think what, what's also important here is that the complexity that's developing around these types of systems with things like agent based systems, the hybrid architectures we spoke before, make this a really complex thing to manage where you are not even, uh, clear in some instances how the models are working together, how different models are interacting.

And you raise an excellent point where models are being used to generate, uh, code, how is that code verified, uh, and actually deemed to be safe before it's implemented, especially where you have, uh, automation within the, the types of frameworks that might be developing that code. And then the third thing I wonder about these days is, well, models are built using software and a lot of times they're built using open source software that has well inherent vulnerabilities in it. And do we realize that the bad guys can exploit some of that software to get at the model because the underlying software has, um, vulnerabilities and dependencies that are essentially passed on to the model?

Yeah, and we've seen that, uh, as I mentioned recently, you know, there was the, the published sized, uh, attack on, um, the SSH, uh, architecture using a, a utility library. And so actually that, that was caught really serendipitously. It wasn't, um, by, you know, some, uh, effective process and safeguard that was put around the use of that technology, and that could have had widespread consequences.

So where people are using open source software, I think it's important to understand who is actually, um, curating that software, um, what the effect of that software might be, uh, within the processes it that is being deployed within, uh, and also how you can actually monitor for changes in the software and capture them. And, and, you know, we had the incident just recently with the, uh, the patch issue by CrowdStrike that had widespread unforeseen consequences. Exactly the same thing could occur in a a, an open source, uh, situation where somebody implements something within a model, uh, as I said, it might not be, um, obvious, uh, at, at deployment time, that it has a, a potential downstream negative effect that could be triggers built into there, and that could be either manipulated maliciously or by accident and have consequences that are entirely undesirable.

So I think a lot of work has to be done, um, with regard to the, the safety of ai, uh, system deployments based on, you know, that the software bill of materials, uh, and there's gonna be increasing emphasis on that in the future, I think. Is this something that a cybersecurity team is able to manage, or is this more like, well, we have to figure out how to get into the workflows that are used to build the models and then the production environments where the inference engines are working. And, um, we've been talking about DevSecOps for a long time, but will we need something that feels like ML SecOps or is it gonna be dev l dev ML SecOps someday?

I don't know. But, um, you know, how does this all come together in your mind? Yeah, well then that's, that's another great question that I think a lot of people in the cybersecurity domain are, uh, are juggling with at the moment.

And the reason for that is that the necessary expertise to critique, uh, model architectures and, uh, changes in the, uh, construction of the, the model implementation are not necessarily native to, uh, those charged with cybersecurity. And we've also seen, you know, just recently, a a race to adopt AI within cybersecurity operations, particularly within, uh, SOC implementation. That itself creates potential risks, uh, because there's a concentration of information within those AI systems that that may be, uh, of benefit to a, a cyber threat act.

So having sufficient expertise and scrutiny of how these systems are being deployed and how they actually operate is important for people to understand the security risks that they pose. And I think that's, uh, a challenge at the moment. One of the things that fornix has, uh, joined is the, the NIST AI Safety Institute Consortium, which is a, an industry body looking at, um, effectively how to, uh, de-risk AI processes by putting, um, measures of risk and, uh, measures of effect, uh, around their implementation.

And I think that's gonna be something that cybersecurity, uh, professionals are gonna have to think about as they integrate 'em within their own processes. And also they try and control the, the operation of these systems, you know, within the, the existing security frameworks and policies that they operate under today. It also seems to me we kind of treat the training of these LLMs is kind of an all or nothing proposition, and yet, do we need to always expose sensitive data to them to get to the outcomes that we want, or can we be smarter about what data we're actually showing them?

And maybe, you know, if I wanna train it, I don't know necessarily need to show it everybody's social security number. Yeah, that's a, that's another good, uh, good point. You know, there's been a lot of interest in technologies like format, preserving encryption and tokenization, uh, and also cryptographic methods for deploying, uh, models in order to ensure information security in their operation.

I think this is challenging. It's, it's an area that we're, we're looking at and working on with our customers and, and, and, you know, conducting research with, um, other industry partners to, to understand the best way forward. And the reason for that is that, uh, fundamentally I think particularly with generative AI systems, they have some intrinsic insecurity in the fact that they work on based on plain text.

So to your point, you know, I think it's, it's worth considering what information needs to be exchanged with the model in order to get the outcome that's required to, to do something meaningful with your data. Do you actually have to show the data where there's a lot of interest and has been recently of things like, uh, retrieval, augmented generative ai, and one of the risks there is that the, uh, model can actually ingest that context that you're using to enhance its performance, and then, uh, you can recover sensitive information that you shouldn't otherwise be able to do from that, that context in the, the output. So there's a lot of issues to address there.

One of the problems with cryptographic methods is that, first of all, it has to be consistent with the training of the model itself, otherwise the model doesn't have any comprehension of those tokens that you are, you are looking to use to mask your data. And the second thing is with, um, obfuscation of, of model function and things within, uh, systems, there's also problems of actual transparency and explainability and how the model's deployed and what it, what it's actually doing. And so I think this is, uh, you know, probably the, the most pressing problem facing people at the moment is how to maintain data privacy with systems that are, uh, you know, to intent and purposes insecure by, by default in the way that they operate and the way that they, uh, they function, um, as, um, token generators.

Right. And you mentioned clear text, and I know you guys are kind of been heavily involved in this whole confidential computing movement. Do you think, uh, AI is gonna maybe drive more people to embrace confidential computing?

And for the uninitiated, maybe you wanna explain what confidential computing is? Yeah, let, well, let's start with a definition. So, uh, confidential computing is, uh, protecting memory, uh, called a trusted execution environment using a, a hardware based, uh, cryptographic key on the, uh, the platform that you're using, whether that be a, uh, A CPU or A GPU and, and soon to be an N ps as well.

And, and what that does is it effectively encrypts the memory so that if you try to extract information about the process that's running there, it's encipher and you can't recover, uh, secret information, cryptographic keys, et cetera. And full X has pioneered that technology, and we've got a lot of experience of using it with ai, including papers that we've published in a number of different, uh, industry domains. I think within the, let's call it the confidential computing community, there is a lot of, uh, interest in how the technology can be used to support AI security.

And in fact, you know, my personal view is that as we move forward to, uh, more complex systems involving, uh, agents and interaction between different domains and different model, uh, architectures, there will be a need for this technology, uh, for two reasons. First of all, it will prevent unauthorized access to the information that's being processed, whether that be, uh, the context that we mentioned before in rag processes or the, um, the underlying, uh, training data within the model. Uh, and it also can be used to place safeguards to sort of intermediate between the user and the model, um, to actually capture information either on the, on the way in by effectively scrubbing, uh, information that shouldn't be transacted with the model under organizational policies, and also then capturing, uh, any information on the outbound side of the model before it's communicated to the user.

So using those trusted execution environments and being able to validate the integrity of the software that is running within them, I think is gonna be important to deployment of AI in the future. And with the new, um, capabilities that are being brought by, um, things like the, the Nvidia H 100 confidential GPU architecture, it's also providing the necessary performance to actually, uh, make security a, a functional component of the, the AI system design. Now, it's not clear to me that a lot of people realize this, but it seems to me, um, no matter your best intentions at the moment, most of these LLMs are overly anxious to please, shall we say.

And so some folks say that they keep a secret about as well as your average 5-year-old. So, and a lot of that just has to do with the people are coming up with workarounds for prompts to, uh, pull out sensitive data. So do we need to also keep track of what prompts are being used and how they're being used?

We certainly do, and I, I think, you know, you, you, the analogy you used is great. You need to think about these systems as an insecure five-year-old. And, uh, you know, if you don't want to divulge secrets that get revealed at a family event, then uh, don't communicate it to the five-year-old.

And, and if you do, make sure they understand the behavior, what you want from them. So, as I said, prompt inter mediation, I think is, is where the industry's at at the moment. There's a lot of systems being built to try and achieve that and to integrate that with policy management and, um, you know, effective controls.

And it, and I think in future this is going to be, you know, the, the, the sort of area of research that's gonna bear fruit from a security perspective is that we're gonna have to effectively learn to work with the LLMs, uh, and generative AI systems as insecure, um, uh, artifacts, and then to, to actually put sufficient governance around them that we can constrain their behavior to the, the tasks that we actually want them to perform. And then of course, you know, as we've spoken about previously, what we need to do is to ensure that in doing that, we're also controlling for what those artifacts are and how they're, they're actually managed, updated, uh, and corrected, you know, through, um, training and fine tuning, you know, once they're in service. Um, I think that the problem is that people are still trying to understand exactly what the, um, the scope and the landscape of the security vulnerabilities is with these systems, how they actually function in practice.

Um, there's a lot of research being done to try and break them with jailbreaking and, uh, prompt injection attacks and things like that. And there's also a lot of research being done to try and, um, uh, make them more resilient. And, uh, you know, that this is still a work in progress.

And I think that the AI that we see in practice, uh, with large enterprises in the future is probably not gonna be, um, that close to what we see today. You know, we've got some foundational technology, but I think there are, um, new, uh, advances that are gonna gonna come in short order that are gonna help to, to provide some of the, um, constraints and boundaries that we need to make these, these things safe and effective. Unfortunately, a lot of the people doing the research in this area are not necessarily, uh, shall we say, good people.

So, um, what is your advice to folks to, as you pointed out, the, the situation is developing, shall we say, in terms of the technology and the use cases, and we're still researching the security issues. How, how do I navigate all this? What should I be doing today to kind head down the right path at least?

Yeah, it's very challenging because, uh, you know, you've only gotta spend half an hour on LinkedIn and you have about 300, uh, archive papers to, to read through and, uh, and about 500 opinions as well. So information overload is one of the problems. How do you filter the, the nuggets of gold in terms of technical developments or, or, uh, you know, methodological approaches from a lot of the, the also ran noise that, that, that's, yeah.

Difficult to do, but I think the, the reference point is actually some of those large organizations, some of those foundational, uh, model developers in terms of the work they're doing because they have the, the resources, the skills, and the, um, the expertise in the systems to actually, uh, communicate validated information. But I think there's also a strong role here for academia and, and peer review. And I think, you know, what's interesting is that, you know, whereas there's a lot of, um, research going into the, the foundational models themselves and they garner the most attention because there's a lot of money behind these companies and a lot of interest in new capabilities.

Some of the, um, sort of more peripheral research around things like, um, data privacy, uh, security, uh, uh, and things like that, uh, that's being communicated at some of the, uh, the mainstream academic conferences around AI and machine learning is actually very interesting 'cause it sets the direction, uh, in terms of the, the thought we have about how we should be, um, looking to manage and control these systems. So I think engaging with, uh, the literature, uh, you know, that's been peer reviewed within journals and conferences is important. Uh, not just, you know, what you see in blogs, but, but also equally keeping abreast of developments that are happening within the, the main foundational model providers.

'cause effectively they're gonna set the trend for, uh, the industry moving forward just on, on the basis of their resources. All right, folks, you're hearing it here. Well, the good news is AI Securities a work in progress.

Bad news is AI Securities a work in progress. So stay tuned. Hey, Richard, thanks for being on the show.

Thanks, Michael. That was a great way to end it. Appreciate your time.

All right guys, back to you in the studio.