Trishank Karthik Kuppusamy – How to Build a Compromise-Resilient CI/CD

And how to build a compromise, resilient CI/CD pipeline Trishank Karthik Kuppusamy of Datadog will show us how they develop the industries first end to end their pipeline. That automatically builds integrations with a Datadog agent. That's like.

Hi. Hi, everyone, I'm Trishank Karthik Kuppusamy. I'm a staff security engineer, Datadog, where I work on things like how to build a compromised, resilient CI/CD, which hopefully will make sense in a few minutes.

So let me set some background, some context and motivation for this, why are we talking about this? So Datadog, as some of you might know, is, is an observability, a platform for your organization, spends all the way from your network infrastructure, all the way to application and services ultimately to end users. Right.

And so when something goes wrong, let's say there's a latency problem, for example, your dev ops, if security and your business now all use the same tool to diagnose exactly what went wrong and fix it on the spot. Right. And so the agent is this piece of software that you typically install in your containers and to together metrics about your services.

And you can think of integrations as plug ins or add ons that add superpower's today and as it were, lets it observe even more. So, for example, one of our favorite integrations and is GitLab. We're hoping you're using both today.

And so what happened was that a year or two ago, we had this problem where we wanted to decouple the integration's, the release of the integration from the Asian, because the way we typically release them was every six weeks together with the agent, you can imagine disproves the problematic. So, for example, what if we wanted to beta test new versions of integrations with our customers? We couldn't do that so easily.

So we wanted to decouple them. And of course, the state of the art for doing this is use something called CI/CD, one of the most famous examples being GitLab. She's done a wonderful job with pipelines and so on.

And so it's I like to joke. So now you use this robot sitting in a cloud in the sky. It's got access to your code signing keys.

Every time your developers check in code, this robot takes out the key builds the package right. And signs it and puts back the key and releases the software. Wonderful.

So if on demand you have dev ops now, basically. Right. Which is great and also good security reasons for doing this, actually, you'd rather have a single point of auditability and login where you know who's using the key when, for example, a robot as opposed to a distributed team of developers.

So great. That's the state of the art CI/CD, which is very good. So let's talk about what can go wrong, because this is a security dock after all.

They have to be optimistic here in good times, ninety nine point nine nine nine nine percent, whatever. Percent of the time, everything's good. Nothing goes wrong.

Life is good. The problem is, is when any of these pieces get compromised. So, for example, imagine someone runs away with a developer signing people up saying get commits, for example, or your GitLab repository, for example, gets compromised to someone temper's WhiteSource Google.

Or let's say you're GitLab pipeline gets compromised, right, the runyon's, for example, or the container registry used to pull images used to run the GitLab jobs or when your key or file servers get compromised. I think you get the idea. And sort of belaboring the point here, the point is that there are many pieces that can go wrong.

And the thing is that it's sort of a great one. It's not even a black swan problem. A black swan is an unknown.

No, no, this is a no, no, no. It's not a question of if you will get compromised, the question of when and you want to be prepared for it, because like I said, ninety nine point ninety nine percent of the time that it goes wrong, life is good. That rare event, it happens.

It's negative infinity. That's the outcome. And so the state of the art is we have dev ops, which is great, but we're missing Dell seconds here.

Let's talk about how to fix the. So let me try to convince you that this is not merely some sort of a Hollywood bad hacker sci fi kind of movie kind of scenario. This is actually problems that happen in the real world.

Right. Let's take a look at horror stories. One, as you might recall, is the flame malware from 2012 not too long ago.

What happened was that someone, many suspect it's a nation state attacker, pretended to be Microsoft. They wanted to get to Iran to dismantle their nuclear program. And what they did was a break, unfortunately, a hashing algorithm called MD five that was used back then.

It had theoretical weaknesses, but these attackers were where they were found, a novel cryptographic attack on edified and even academics had not seen before. And so they produced the fake certificate that looked like it came from Microsoft and propagated this update all the way to Iran, where it cost the centrifuges, centrifuges to every once in a while very suddenly messed up. So imagine this is very high, very sophisticated attacker.

And depending on who you ask, this was a good or a bad thing. We're not blaming the damages here, certainly. Scary, scary, number two, more recently, see CCleaner, which is a popular windows clean up, do I remember using it, the young men myself many decades ago, while they were busy being acquired by Avast, the security company famous for making antivirus software, for example, while this acquisition was happening, unbeknownst to both of them, someone had compromised the ability to structure.

And used it to build malicious code that was then sneaked in to see CCleaner million to download, but they were interested in 11 or so particular companies, one of whom leads to a scary serial. Number three seems to have been ASUS because they got their own software updates back out later. And the suspicion is that they got to ASUS by poisoning the well so well and then they made their way to ASUS.

So we're not playing amateur. We're playing a very serious professionals, nation state attackers even, who are interested in breaking into CI/CD pipelines, something you should think about very seriously. And so as a response, some might say, well, why not just use GPG or TLS will not solve the problem.

No. Remember, we're talking about nation state level attackers. They're not going to be fazed by a single key used to sign everything kept on infrastructures.

This is like the equivalent of keeping the keys to a house under the carpet. It's not going to buy you the security that we're looking for here, unfortunately. The property that we won is something we call compromise resilience.

What does that mean? Well, imagine if you're a medieval age king and someone told you, hey, we know that people breaking into fortresses is a problem here. It's a very common problem these days.

So but don't worry, we've got the solution. We'll build an impenetrable fortress. And so the king should be very suspicious.

It's got a snake oil salesman. No such thing. I know there is or it would be prohibitively expensive.

The point is that you cannot prevent a compromise because if that's infeasible, if not outright impossible, what you can do is to mitigate the impact of a compromise. So you build defense of death and security. You have multiple layers so that attackers you have, for example, a moat surrounded alligators and flamethrowers and whatnot, and you get the idea.

Point is, attackers look from far away and say, you know what, I got better things to do with my time. I'm going to move on to another dark. That's basically compromise, resilience in a nutshell.

And so how do we do this? Well, we're going to propose using two pieces of software to be technological in total and tough that I can get to this property. There's this joke.

I don't know how many how many of you have seen this vitamin water ad. And it's four am do you know where your vitamins are. So I've just this joke for my own nefarious purposes to say it's 4am do you know where your CI/CD is.

m. and says, you know what, it looks like a pipeline released, something malicious. You know what happened?

Let me tell you a story. So I used to be a developer and software company, one of the very few that had the privilege of being audited by the FDA. You know, why is because we made software for Big Pharma to run clinical trials in very serious business.

So we needed to have the ability the auditors could walk in and our randomly 10 years later, then use a version of software and they would say, show me the output of your unit and integration that show me prove to me that your software did what you claim, that the fact that. Right. And we won sort of the same sort of auditability we want the same sort of compliance for some of you may even be a compliance requirement.

So you can think of a CI/CD like this. X is the source code that you develop, produces and you CI/CD is basically applying a function F to X your source code and produces a package Y, and what you want is to is the property that when you download and install the package before you install it, you say does Y equals F of X is the correct X, the source code is the correct application for packaging of the source code and do the package I'm looking at right now. Right.

So that's the basic idea. And how do you do this? Well, one piece of the puzzle is in-toto, which secures the distribution of your good packaging.

It a pipeline rights all the way between the developers and you're in your CI/CD. And tough as it is to do, that's all. The other half of the problem, which is just to secure the distribution of your packages from your package, your boss read your end users and you put the two together, I claim you get the practical end to end security, which lets you deduct that tax a bit between your developers and end users.

So let's take a look at how that works. Unfortunately, I don't have time to go into all of the gory technical details, but here's what you need to know. We know that the problem exists.

That's the bad news. The good news is that there are tools that you can use today to fix the problem they call in-toto and TUF. And that's all you need to know and don't want to get you compromised or.

OK, so let's talk about the first piece of the puzzle in-toto, which, remember solves the problem of detecting at anywhere between a person and CI/CD the problem that in-toto solves is kind of a nice metaphor is the game of telephone that kids like to play. So imagine transmitting a source code CI/CD all the way to end users, for example, a sort of like playing the game of telephone it kids like to play. Used to be fashionable, I guess.

I don't know whether they play it anymore, but imagine that attackers got in between these kids and tempered the messages that sort of how in-toto works and tries to guarantee the authenticity and integrity of these messages, what is being passed. So, for example, the first kid here say and a second kid slowly the message sources of noise, which you can think of the attacks, unfortunately, fleas get delivered there and it's a bad idea when you want to deliver peace. So let's try to fix that problem.

How do we do this? Well, the basic idea is to define the software supply chain, right? You remember the F of X problem.

This is basically what you're doing. You're saying this is my F effect's. Here's what my F of X may look like.

So Alice, who's the administrator, may say, my supply chain looks like this. It consists of two steps, but Diana is allowed to write source code and she produces a sign at the station saying, I promise I produce. And the second step is Bob, who's allowed the package to source code that Alice wrote.

And Bob says, I swear I got through it all by with this hash and I produced a package. I simply compressed the file and server the users and I produce the tar with this hash. Great.

Now what happens when that and uses before they install any package, differenced inspect it. And all of this is done transparently using another about this being taken care of behind the scenes for you using software libration. What happens behind the scenes?

The Alice can say, look, before you install the package, make sure you get to sign at the stations from both Dianne and Bob and of course, the supply chain itself from Alice. And check that all the rules that that F affects is indeed the case. So, for example, here, is it an inspection?

gz Is correct? What would Bob produced? And then you enter the file and say, OK, I was the one that Alice produced and there was no attack in between C, so no one other than Alice can tamper with the source code.

And so that's how that's how you get the supply chain integrity. Now, let's talk about a second piece of the puzzle with just the update TUF for short, and that solves the problem of the so-called last mile distribution, which is the distribution of the build source code here and uses. And the metaphor, the problem is a tough so you can roughly think of it like this.

So I don't know if you remember that back in the 80s or 90s, it was a terrible, terrible attack in pharmacies where someone malicious obviously went around and deliberately tampered with medical drugs, people's medical drugs. And unfortunately, a few people were poisoned. And I believe that even actually died, unfortunately.

So you can think of stuff. What it does sort of in total is the thing that tells you, hey, my software is a list of ingredients. So Diana produce this ingredient with this dose and Bob produce that ingredient.

It just goes and the whole medical drug is composed like this that does that those and you mix it together, put in a nice little pill stuff is the software that says, why should you trust this medical drug in the first place? You see it thing. It's like the seal of freshness, authenticity and the hologram sticker that say, yep, FDA approved, good to go.

Nothing has been tampered with. That's the rough idea. And so I don't have time to go into details, but basically use design principles that, believe me, a grandma would have told you as a kid about your grandma.

S. military. That's, for example, is that actually a physical separation where not only are two kids required, but the same person cannot turn the two keys together.

You literally need two different people and the same idea of the threshold signatures here. So we also use design principles. Grandma told me, don't put all your eggs in one basket.

So that's why you see multiple eggs all over the place here. Grandma also said make sure you use cryptographic agility and that's the story of the Hydra. So.

So, for example, remember the flame malware attack that happened because unfortunately it used one week hashing algorithm by MD five and we use multiple hashing algorithms with different designs. So 2 and 3, for example. So unless you can break them both.

You're not going to be able to break the security system. So anyway, you put all this nice design principles that grandma told you, and we actually designed this in collaboration with Thor, who wanted to obviously protect the software updates from people as five of the nation's attackers. OK, enough theory, let's talk about practice.

How do we actually apply to and in practice for the Datadog agent Integration's, remember, was the original problem we were trying to solve. And I claim that putting the two together, you get end to end security. Ok, so here's what our software supply chain looked like earlier, we looked at a demo.

Let's let's take a look at a real life example. So I was always, by chance got three steps, the first one is called TAG, and I get that where developers use Yubikeys is sitting on the Yubikeys and trusted hardware so you can't even export the keys. So they find the sign at the station saying, I produce the source code with this hash and I'm checking it into into our git repository.

Right now. It's CI/CD is broken into two separate days. You know, I swear I got this python source code from the repository and I'm packaging it into a python wheel, which is simply a zip file.

gz, and it packages the Python source going into the zip file. And then we have a third step. We'll sign that which basically takes this packages and in total metadata puts it together and it's tough seal freshness and authenticity and integrity and distributes it in a nice, coherent package to end users and our end users and solve this.

They have no idea that all the stuff in the verification actually happens behind the scenes. What happens behind the scenes is that we tell the agent, hey, look, first make sure that the wheel is produced by the wheel are great, and then unzip it to make sure that the source code was signed by our developers. So you can see that unless you get developer signatures, even if you break into CI/CD developer key, sorry, if you break into a CI/CD, you won't be able to forge all the signatures, which is where we get the end to end security from.

Ok, and then we add another picture looks complicated, but really simpler than it looks like. Basically with solving three problems here, what we're using tough is to say, first of all, we we deliver when there are many keys to the entire system, including distributing developing keys. Think about it.

How do you safely distributed developing the software supply chain? That's what we use that for the one root key that we distribute the agent and then you can change transparently, rotate the keys and the rest of the system. The end users wouldn't even know.

We know because we've actually done it several times. And so what we use stuff for is to distribute the software supply chain in a compromise resilient way. So just because someone breaks a pipeline, they won't be able to rewrite the supply chain.

They won't be able to rewrite. The public is used to verify the supply chain. And even though the machine and those are the things colored in red, signing some things, they're not signing the crucial bits.

They won't be able to mess with developer signatures without being caught. That's all you need to know. That's the level of detail you need to care about.

If the interest in the more gory technical details of this, visit, this link here. Ok, so what does all that gobbledygook get us? Let's see.

Well, nothing has gone wrong. Clearly nothing. But I think it's important to also point out that your end users, actually our end users actually don't see any difference with or without TUF and in-toto, which is to say that we add very little performance or network overhead to the security system.

So most of the complicated, the the setup, the price of the setup is in one time set up the whole thing and then the system basically pretty much maintains itself after that. OK, well, let's see whether with a real beauty system like this one is OK. So if I were developing is a compromise, yes.

Theoretically, what can happen is that attackers can release malicious code. But remember the compromise resilient property I was talking about earlier, medieval fortification. We believe we've set the bar so high that this is actually more theoretical than practical.

Let me explain why. First of all, our developers signing keys are trapped on you because we generate them on the hardware. We never export them.

We are unable to export them. And so unless you physically attack one of our developers, please don't do that. You won't be able to get the you won't be able to get the keys.

OK, that's the point. You can't remotely run away with the keys. And the second thing is that we require you to touch your YubiKey every time you do a signing operation.

So even though this malware is sitting in a developer's laptops saying, hey, you should sign me up, our developers would say, wait a minute, what's going on? I'm not signing anything right now. This is the third thing that we could do.

But we've not done for the sake of usability right now require at least two different developers to sign off on the same source code. Right. This is so that developers can more easily release source code, but you can see how it can easily increase the security of the system without actually hampering usability too much.

So in practice, the Tubikeys are very, very unlikely to get compromised. And the beauty system comes through the rest of the way, actually. So what happens when our GitHub repository in this case but could be GitLab gets compromised?

I think we don't lose sleep. In fact, we've seen accidental DOS attacks where developers sign mismatching versions, basically with the GitLab pools, doesn't match whatever developer signed. You do not not merging branches properly.

And so it looks like an attack, but it isn't. So we don't have to be very, very good tests of this thing in practice. And the point is that even if our good repository gets compromised, we don't lose because the attackers won't be able to propagate.

Basically the download. It would block it, as you see here. What happens to a GitLab gets compromised?

Same thing. We don't lose sleep. What happens to contain image registry that GitLab uses gets compromised?

Same thing. Don't lose sleep. What happens to our key servers or file Servers get compromised.

Same thing. You get the idea. The point is that a download this transparently on behalf of our agent verifies this stuff.

And in the meditate at the moment it smells a slight of suspicion of an attack, basically denies installation of the package. And I should mention I should take the pains to mention that as far as we know, this is the first industry we haven't seen any public discussion of any similar system. This is the first compromise, resilient, at least the publicly discussed compromise, resilient CI/CD that we have seen.

And it's no note that there's no trust in hardware here except for Yubikeys the cloud is perfectly untrusted. You don't need trusted hardware such as in Dell and similarly complicated trusted anglaise, which have their own security bugs these days. Right.

All he needs is something like you because in fact, you don't even strictly need it and you have this very high level of security. So so the point is, as I said earlier, the bottom line is there is a problem. Yes.

Which is the state of the art CI/CD practices. Unfortunately, typically we don't build compromise, resilient CI/CD that's the problem. But the good news is there's two pieces, technology, open source technology, both of which are on the NCF, which you can use today to secure your own GitLab pipelines, and they're called in-toto and TUF and gives you End-To-End security anywhere between the developers and users.

I should mention, should takes some time to mention that all of this work wouldn't have been possible without some great people at Datadog, NYU and VMware, and I don't have the time to personally shout out to each of you. But you know who you are. And thank you very much.