AI Security is Not a Pipe(line) Dream | RSAC Virtual 2024

Hello, I'm Frederick kz and I'm here to talk about AI security. So before we start, uh, lemme give you a little bit of my background. Uh, so I co-authored the NIST S 802 0 4 D guidance on software supply chain security for DevSecOps, uh, written, uh, several white papers including the cloud Native Security white paper.

And I've also have co-authored a book. Um, I have extensive experience working with, uh, with ai ML and a variety of their, of other technologies around Cloud native. I have architected large projects that have worked in highly regulated environments, um, and I've also have experience working with standards groups, uh, trying to work out how to provide guidance to, to other groups through, through standards.

So, in short, when it comes to AI supply chain, uh, this is an area that's still developing quite, uh, quite significantly. So when we say, well, what is an ai AI supply chain? Uh, so generally when we talk about supply chains, we talk about everything that can have an impact on either the quality or security, um, of the products that you're developing.

So in terms of ai, uh, starting from the very bottom, you have hardware. You look at source code, um, AI supply chains also have, uh, some, uh, really interesting, uh, really interesting areas around data. How do you defend the data?

How do you know what data was fit for use? The hyper parameters you decide to use, uh, all of your dependencies for your source code, your model dependencies, whether you're bringing in another model to fine tune on. Uh, all of the processes around it are also part of that supply chain, because even if they're not performing an actual, uh, function within your, within your supply chain, they're still controlling and ensuring that the things get done, uh, and so on and so on.

So, when you think about AI supply chain, it's quite extensive. Think about everything from star to end for the whole process, from, uh, we're starting with the first line of code all the way down to this thing is being deployed in production or even being sunset over time. So how do you secure everything then, because that's a pretty big ask.

So what we tend to recommend is to, we start to break it down into steps, and we reason across them, 'em at multiple levels. So the very first thing is saying, well, what, like, what are we trying to accomplish? And what steps do I need to perform in order to accomplish it?

Uh, you want to define what the, the process for like, what all of those steps together. And after you, after you put 'em all together, you want to secure each step first in isolation. Like, how do I secure my build?

How do I secure my, uh, my model storage? How do I secure that? I'm po pushing this thing into, into production?

And those you can also break down into in their individual steps as well. We also, so once we've, uh, tried to secure each step, uh, then we want to secure the interactions, as in I'm pull, I'm pulling some data from somewhere that's getting stored somewhere. Maybe it's gotten normalized.

So now that I have this normalized data, my model, my my model trainer, my needs to pull it in so that it can go build the model. So how do I secure the interaction between the thing that, uh, perhaps where the innate data was initially stored to where, to where I'm actually going to use it. Um, and after you've secured the interaction between the individual steps, then you can look at defining and securing the workflow throughout the whole pipeline.

As in from very start, I have some data that's, that's coming in, uh, through some means, and that goes through the whole process of being, of training a model and so on. And it goes all the way down to deployment. So how do you, how do you secure end to end and make sure that the things that you need get to where they need to go, and you repeat each of these on a regular basis to make sure that they're still relevant, that they're still operating in the way that you want or you find issues with it.

How do you resolve those issues over time? Um, and of course, that doesn't mean you do all of these in the exact order that's here. Uh, it is just showing that you break it down into smaller steps and you try to identify what's important.

What do you not want to happen? And you take, you take steps individually for, for each of these particular, for, for each of these particular steps. So, um, AI pipelines and software supply chains turns out they have a lot in common.

Um, they, the area they have in common the most is first, they, they all use software. They, the actual CI CD systems tend to be different, but they all have the same set of principles. So if there's some pipeline, your information is flowing through that pipeline, it's performing a transformation.

Something's coming out on the other side. Of course, there are exceptions to this in terms of, uh, in, in, like you look at federated learning, but even federated learning has steps that you perform. You have, uh, you're trying to control for risks as to what happens when you're not in control of, uh, of the, um, of the learning step as, as closely as you, as you, nor as you would be.

So even this still has, uh, similarities and that there are a set of steps. You're gonna perform some action, and you want to make sure that those things happened. Um, and AI pipelines tend to have some added complexity in that they tend to deal in many cases with very large volumes of data.

It's not uncommon to hear terabytes or petabytes worth of data being consumed. Now, throughout through such a pipeline, they tend to be very, uh, sensitive in certain, in certain scenarios where the information coming in has to be controlled. You have to go through your, through your data owner, you have your data custodian you have to work with, and you have to make sure that you are protecting the models because the models themselves can learn information about this, about that sensitive information you're training it on, of course, all the privacy concerns that come with it.

So there are real complexities that are above and beyond a normal software supply chain that, that enter into this, that need to, need to be taken into consideration. Um, but even then, the, the principles are still the same. So, um, part of, uh, part of what we also wanna look at as well is since we have that data, um, perspective in there, we also wanna make sure that we're looking at data privacy, then model integrity that, uh, the type of actions that we're taking are indeed representative of what we want them to be.

And, um, if you ha, if you're training on sensitive data, you, that you're using the appropriate set of techniques. So if you're not applying, uh, if you're not applying things like differential privacy or other similar types, types of, of techniques, then, uh, how do you, how do you protect the model so that the model itself, uh, you prevent it from being leaked out so that you don't have, if somebody manages to, to, to get ahold of that model, that it, um, you wanna prevent someone getting ahold of the model so that they, you don't get model inversion attacks where people are trying to replicate the original data set from, from the model. So one thing that you, that you should not overlook as well is the importance of human processes.

Because everything I mentioned before or originates from human processes, people working well with each other. Uh, at the very top, if you go to any organization, you're gonna see a set of policies, you're gonna see standards, you're gonna see procedures, guidance, you're gonna see regulations that, that guide all of that. Uh, so you want to make sure that you're fitting in with those policies, that you're not just throwing everything out of the window.

Uh, and, uh, and then just, uh, running at full speed towards a spot that's gonna lead to, uh, towards a loss of, of data, loss of trust, a loss of, uh, a loss of material. And part of this as well is making sure that all developers and people who are working with the system are aware of what their responsibilities are. And, uh, these are where things like incidents, response plans, um, regular security audits and similar start to really take place, uh, please do not, uh, underestimate the value of, of working with people in order to, uh, in order to help them do the right thing.

So one of the frameworks that we use in the software supply chain space to do this is a system called in Toto. So in Toto is a framework that we use to secure to help secure software supply chains. And what it does is, this is a lot of people, or many of you may be familiar with software, Villa materials, where you list all of these pieces of software that you are using to build a, uh, to build a product that, um, in terms of what in Toto does is it takes the process, like I have a bill step, I have a testing phase, I have, I'm delivering this to a particular location.

So you list out each of the steps you're going to perform, and then the in, in total will observe what is, uh, what is occurring and will then produce a result that is in the form of an attestation that demonstrates that it, it's following the plan, that nothing extra has been, uh, added into it. And that the, um, and, and that every step is, uh, that you've taken, that you have observability into what, uh, into what the result was. So you're looking at, uh, creating a layout.

You're looking at metadata, you're looking at rules around how, how you want those artifacts to be treated like. These are the kind of things that you can put into in toto and, and write policy around. Um, and so the, the mechanisms of in Toto is you start off by creating a supply chain layout with the authorized, they call 'em, uh, functionaries or basically like, this is a particular function that I am going to allow to perform at a given step.

And you collect that metadata. And so one example is like, we want to verify the integrity of a software package before we deploy it out. So that means checking the signature, checking any hashes, making sure it's the latest version.

Uh, so, uh, basically well spelled out really well so that it's, it's very clear as to what steps need to be, what need that, what steps need to be taken. So applying this into ai ML you can see here we have a download phase where, uh, this download phase could be like a data collection phase. It doesn't have to be download.

This is a simplified model, of course, where okay, we're gonna create, uh, some information around data integrity, and those attestations are gonna be put out so that we're able to go consume, consume them at a, at a later date or a later time. So now we have multiple, uh, now we have a model building phase of, and there may be multiple model building phases that are independent of each other, that all consume from that initial download or data phase where you're now performing some form of data processing. You're capturing the information about how that occurred going into your model training, you're creating adaptations about how you trained it.

And all of this then goes into, all gets preserved, uh, all the applications get preserved, and then you send it off to the model to towards your deployment phase where you're, you're validating, excuse me, you, you're validating information about the model that you're, that you're using. Is it the right one? Is it, uh, fit for use?

Um, are we, has it been corrupted in, in flight? Has it been tampered with in flight? Um, and then you collect information on that as well.

Um, so when you're collecting all this information, you wanna send them, you wanna send these into somewhere that, uh, they can be observed. Um, and ideally, every single one of these attestations will be signed, uh, as you, as you create them, so that it prevents tampering at, at a later time if a, uh, if a threat actor enters into your, into your system. So a couple key areas where in Toto has seen significant adoption is, first, there's the, uh, salsa, uh, supply chain levels for software artifacts that, uh, provides end-to-end software supply chain security.

It's a specification that's provided by the, by the open SSF, which basically says, if you want to build a, uh, if you want to build a software, we call 'em software factories, then these are the kind of things that you should have in your system to help defend against common, uh, uh, against common threats. Uh, is it's, it's very well laid out. It's designed to be generic enough so that you can, uh, you can apply 'em to multiple types of systems.

Of course, you need to fine tune it and make sure that, that those steps are every single one of 'em is relevant and make sure you're not missing anything if there's something, uh, unique about your environment. Uh, another one is NPM. So, MPM, uh, is in the process of building out a system where, uh, you can run MPM Verify, but what MPM Verify does is it, um, when you do mp, when you do a MPM build, if you do 'em through a, like GitHub actions, it could be, uh, through GitLab or similar that they create a salsa attestation, which is actually, uh, written in, in Toto that has a couple pieces of information about the process.

Uh, so first it says, where did what commit that it use in order to generate this particular, uh, in order to generate this package that the build was, uh, was performed on a system that the developer does not have direct access to? So they can affect the inputs as what goes into it through the, through the GitHub, um, or through the gi, uh, commit, but that they have no ability to interact with it after that. And whatever the result is, gets captured, analyzed, it gets hashed, it gets signed.

And when you do and like that, then the MPM package when it's uploaded, has these additional attestations that have been signed by this third party, uh, that guarantees that that commit is the originator of that, um, of that package. So it prevents people from just, uh, doing MPM publish to, um, um, from their desktop where the thing they're publishing is not, uh, related to the thing that they claim is a bond git. Um, and so it, it helps, it helps reduce the total number of, uh, of attacks and also makes for a much more auditable system, because then you can, you can now go and look at those particular pieces of information.

The same things, uh, can occur within an AI pipeline, as in where, where did this thing come from? What were the inputs? Uh, making sure that only a system that's authorized to actually perform a, uh, a model build has performed that model build.

And it was not a model built from someone, someone else who got injected into your system. So, um, in order to extend, um, extend the, uh, functionaries, uh, not in total itself, but the functionaries around it, we have to look at how do we handle the inputs and outputs of, uh, of data? How do we secure the data flow and the model lifecycle?

How do we verify the provenance and the, and the model updates over time? As in, how do we make sure that, uh, the data, that's not just where did the source code come from, we have to ask, where did the data come from? Did it follow all of the relevant processes?

And example of this for, uh, this one is for, uh, is just to show what a functionary looks like. So we have steps in steps we have, and we give it a name. We're in the scenario, we're cloning a repo, uh, and we're providing the expected command.

And so this is like the part of the inputs of, of in toto. Uh, then there's a, there's a build phase. Um, and this one is purposely kept simple in order to, in order to make it understandable.

Um, and you have like a test phase where you're saying, this is how we run the test, and these are the expected products, the expected things that we, that we want to see come out of it. And so, and in toto system would consume this, and then would perform this and create an attestation that demonstrates the, the output. So now that we have that, uh, let's jump into actually validating, uh, data attestations.

So Terrapin is a project that I've written. I've, I've so far created two reference implementations. One is written in go, it's the more mature one at this point.

Um, maybe by the time you read it, uh, they'll be, uh, equal with each other. And what is designed is for, there's a specific problem when you're trying to hash large quantities of data so that you can validate them at a later time. So if you use shot to 56, uh, and you decide you wanna ha to hash several terabytes or petabytes worth of data, um, you have to start at the very beginning.

It's a linear process. And, uh, you go all the way to the end. If you have any interruption, if you have any problems, then you have to restart over.

Uh, this also is true when you're verifying, uh, because a hash is going over the original data, you have to go, you have to touch the whole dataset. And so this is designed to work around some of those limitations, uh, but still provide you with an overall hash, so you can use in order to verify your data integrity to, to, um, to make sure that that providence is, is still what you expect it to be. And so the way that it works is, it, it, it verifies very large data sets by handling the data in chunks.

So each piece of data, each large piece of data gets split into much smaller pieces of data, and each of those gets hashed individually, and then they all get concatenated together. And, uh, then there's a hash on top of that concatenated list that we then generate. And, uh, we can do this recursively for extremely large data sets.

So the way that this ends up, uh, looking, is something like this. So at the very top, I have a very simple example, like an index where, where Shaw hashing the, uh, SHA 2 56 over the first two megs of a, uh, of a, of a block. And the reason we chose two megs here is, uh, uh, so part of my background was in, uh, creating, uh, storage systems.

And, uh, part of it is you want to get a good alignment with, um, with what many systems out there use. Uh, two megs is a reasonable compromise that works relatively well with most systems. We may make this configurable at a later time to say exactly what the, the block size is, but for the moment two megs is a, is a reasonable, uh, compromised, uh, for working with multiple systems.

So we have, uh, an index. This index gets created where each of these hashes, they have hash of block one, hash of block two, and so on and so on, they each get added in linearly into, in, into this index. Um, and this means that when we look at a piece at a specific block, like if we wanted to look at, uh, there's say 23 terabytes into a large data set, we know exactly where in the index to go.

We know where to, uh, we know what pieces of data we need to read, and we only read the blocks that we actually need to use in order to validate the, the hash. Uh, that index in itself gets hashed, and then we sign that hash so that it, um, 'cause the hash will tell you whether something has been corrupted or not, or whether it's been changed or not. But you still have to protect the hash itself.

And the way you protect the hash is through, uh, things like, uh, PKI, where you signed the, the top level hash. And now that's something that becomes much more difficult to tamper with because you need to know the private key in order to, in order to tamper with it. Uh, so in a nutshell, this is the architecture of, um, of, of terrapin.

And so the next step from there is once, um, uh, once we've integrated, um, once we've created a terrapin hash and, uh, a couple other properties with it, is that because it's running in blocks, that means you can also run them in parallel with each other. So you can throw large clusters at it, and while you're generating the initial hash, or you can, um, you can throw you, you can validate only the parts you need. So if you're running, uh, parallel, parallel systems to validate, to create a model, they only need to check the things that they need to check.

Uh, so what we do is we start with the, uh, we start with in toto, we put that hash within in Toto, and that in toto, the in toto applications themselves are signed. So that provides you with a signed hash that represents that particular data set. And when you move on to the next process, part of the process, you're able to go pick up that top level hash, you go pick up the index, you're able to validate that index, and that gives you the ability to, to validate your whole data set in a, in an efficient way while you're making use of that, uh, model.

Uh, so in conclusion, um, it's, um, please consider the importance of securing not only your software supply chains, but also your AI pipelines. Uh, I would actually argue AI pipelines are, uh, just as sensitive if not more sensitive than a software, than a normal software supply chain, because now you're mixing the data into it as well. Uh, consider things such as in toto, it doesn't, it doesn't have to be in toto.

You can pick something else if you, if you want, but choose, choose something in order to help define your process integrity. And that way you can, gives you the ability to, to demonstrate over time that you're getting, uh, what they call continuous compliance, that you're, that you're always doing the right things and that it makes it more difficult to, uh, to avoid a, an important step. Uh, consider the role of your hashing algorithms such as terrapin.

Uh, there's others I'm sure will come along as well that can help with this as well. So consider the, um, how do you secure very large quantities of data and make sure that the, that your integrity has not been compromised. And, uh, the call to action here is please implement these tools and practices or similar kinds.

Please consider the security of your AI pipelines. Uh, it's not enough to look at the AI safety like, or the biases. You have to make sure that those, uh, that, that those tools and techniques that you're putting in place are indeed being followed.

And when you load a, a model up to run in production, uh, ideally you want to be able to demonstrate, um, and perhaps even gate, whether you run those models based upon the presence of, of those, uh, techniques within, within your pipeline. Uh, with that, I've included a couple references. Uh, first one is the document that I mentioned before that I co-authored for, uh, for nist.

He had me, it talks about this, but specifically for software supply chains. So what I spoke to you here is a, uh, is building on top of that, uh, the in total project, um, which is part of the CNCF, which itself is, uh, part of the Linux Foundation. And we have the, uh, the Terrapin Go implementation and the Terrapin REST implementation, uh, um, that, um, and in both, in both scenarios, uh, the, the SHA hashes with Terapin Go and Rust are, uh, are using, uh, matrix operations or using, uh, SHA one intrinsics when available.

So, uh, they're designed to be very fast over very large quantities of data. Uh, with that, I wanna thank you for your time and hope, hope to see you all at a later time. Please, uh, reach out to me if you have any questions.