Dan Garfield, Codefresh | DevOps Experience 2022

Hello everybody. Thank you so much for coming to my talk. I'm excited to talk to you about securing your software supply chain using Argo CD open policy agent open gitops.

Tyverno and a project called Sig cosine. And this is I titled this talk secure by default with gitops because this is a way of creating a kind of policy driven approach to security where you don't necessarily have to check everything as part of your kind of build process because you have mechanisms in place. They're gonna make sure that things are enforced at every level and one of the one of the deficiencies I think of CI/CD in the past has been that you have maybe policy engine as part of a pipeline but then not a way of actually enforcing that beyond that.

So if somebody were to bypass the CI/CD pipeline they get around it. So this is gonna be a default security creating a default security using these really Cool Tools, all of these are open source tools you can use so to kick this off. Let me introduce myself.

My name is Dan Garfield. I am the co-founder and chief open source officer of codefresh. We are a software delivery company.

We are the Argo Enterprise company. We have a lot of expertise implementing gitops implementing Argo implementing CI/CD implementing software delivery. We've been around for about five or six years and been very proud of this team and what we've built so definitely check out our platform because we have some really great tooling for you to use to deploy software and deliver securely and at scale making your developers productive all those kinds of things.

You can follow me on Twitter at today was awesome. And then I am a maintainer on the Argo project now truthfully. I don't write that much code every once while I do a little bit of code contribution.

Most of the work that I'm doing is helping organize and and organize contributions from our developers helping organize in the community helping build our community events and all those kinds of things. So while I do contribute every once in a while in the code base, most of the work that I'm doing is in the community side of things and on helping organize work and build governance and all this kind of stuff. I'm also a maintainer co-founder and co-chair on open gitops and they get off working group.

So this is a group that we created on the cncf to create standards and principles around getops. You can check it out at open gitops Dot Dev So with that. When I was a kid.

I had a little bit of a side business. Fixing computers set it up networking doing these kinds of things and one of these jobs actually paid me in a laptop. And so for the first time as a teenager, you know, I couldn't drive yet.

I had a laptop. I had mobile Computing. It was a blew my mind battery would last about 40 minutes.

And so what I would do is I would go out on my skateboard and allegedly, I'm not saying I did this I'm saying allegedly, I would go out of my skateboard. I'd put in my headphones with my laptop and I would have it paying and tell me when there were Wi-Fi networks and allegedly I would use my laptop to see if there was a way that I could bypass these and a lot of people if you're familiar with this era there was WEP encryption. Basically, what you could do is you could get in the middle of the of the Wi-Fi router and the clients and you could trigger the real authentication the reissuing of keys and then you could Snoop on that traffic and you could use it to Deal the Wi-Fi password now.

I'm not saying I did this I'm saying allegedly I skateboarded around and did this. So this is not an admission of guilt that could be used in a court of law. I want to be very clear about that.

But what this taught me is that when it comes to security if you can get into the middle of a process. You can usually own the process. So that's that's kind of some of the stuff that we're gonna be talking about today.

We're gonna be trying to solve a couple of problems. So the first problem is do you know where your software comes from? It's eight o'clock at night.

Do you know where your software comes from? And the challenge here is most processes that we have involve us building artifacts pushing them to Registries and then later referencing those artifacts pulling those from Registries and having them deployed. Now, if someone were able to get in the middle and put something and replace your artifact, you might not know about it.

You might deploy something that has a vulnerability into it at the same time. If you bake the image originally and signed it with vulnerabilities. You also have problems.

So that's problem. Number one. So there's actually a way that we can make sure these things are.

Are haven't been intercepted and there's also ways that we can make sure that maybe a programmer who has access maybe their machine is compromised. We can make sure that they can't necessarily push artifacts that are going to be valid and deployed into our servers and this is also a use case that I've seen in the real world in the wild. I've seen a company that had software engineer had their machine compromised.

They clicked on a fishing email it installed a little bit of root kit and then suddenly they started injecting they basically just started injecting a bunch of malicious code into every code base that they had access to. The engineer had no idea so you can actually catch stuff like this with this security approach that we're going to talk about. The second the second question problem that we're going to try to solve is how we enforce rules on our cluster.

So a lot of people approach this with training they try to enforce Automation and they try to have audits. I have really interesting conversations with people sometimes where they'll say things like, oh, we never touch things in production by hand that would be out of process people know not to do that so we know it doesn't happen. And I I love this level of trust that we have for each other.

I mean, it just must be a wonderful organization. So utopian that you don't have anybody by passing the rules. I actually saw a company who had a similar approach years ago.

They actually Unfortunately they had to go through some layoffs and the team that was maintaining a core piece of software had been laid off and suddenly, you know a month later that service went down. So when people went to go debugging and figure out what's going on, they found out that none of the code changes have been checked in over the last eight months. Suddenly suddenly the debug challenge went from 10 to a hundred.

So it's really good to have audits in place and have processes in place that literally don't allow for people to bypass the right way to do things. And certainly you might have some sort of break glass procedure that involves multiple people getting involved, but you want to make sure that that the default path is for people to do things the correct way. That's what this talk is about doing things correctly by default and there's another interesting data point for you a study was done to figure out how they could get more people to opt in to being organ donors.

That's a donating organs saves a ton of Lies. It's super important and if more people donate organs, they they more people's lives will be safe. So what they did is they started saying do you want to opt in And then when they said do you want to opt out the default was to donate your organs?

Suddenly. Everybody was going to donate their organs because if people had to think about it to opt-in they were they were more hesitant than if they had to opt out. So doing things by default it works and the third thing is kind of related get UPS is really powerful and get Ops.

web. You can go through and review these principles now for those of you that are not aware of open get-offs or get off standard or maybe you're maybe you've heard of get Ops it basically has four principles your your state needs to be defined in a declarative format. So most people are using terraform like hey, I checked that box.

I'm feeling great. I'm using yamls manifest things like that feeling great to it needs to be version and mutable. This is where we start to see people stumble because maybe they use something like a floating tag in their manifest.

So it's no longer version or Even though there are used they are checking in to get they're not following best practices to keep those things version unbeatable. It needs to be pulled automatically. That means you need to software agent that is consistently checking what the desired state of Truth is as defined in get and then it needs to be continuously reconciled which means it needs to be aware of what's happening in production and the actual State and then constantly comparing that against the desired State and if you could do those two things and have a Reconciliation policy you can do get Ops and that's fundamentally different than the way that we've delivered software with CI/CD that may be relies on a web hook.

Now you can still use a Web book as part of this process. You just can't rely on the web. So how do we make sure that people are doing this?

There's some great tool training out there that we put out. That is a get off certification using Argos CD course and it has two levels get off some fundamentals and get off the scale. I think we have over 13 or 14,000 students this year alone.

It's the most popular fastest growing training certification in the world Bar None and it is quite excellent. io to check that out, but I wouldn't say that we should rely entirely on it. So the first step When it comes to security training is the first step it is not the last step.

Okay, so we're gonna go into that those other steps that we can take. So approaching security. We want to verify our software supply chain.

We want to enforce security policy and we want to do it all with get-offs. So I'm going to show you how to do this using Sig store cosine Sig store cosine six store is a project within cncf and cosine is an excellent tool. They put out which allows us to sign images to verify their Providence.

We actually just implemented this in the the Argo project. 5 in our go CD those are now signed. So to use this it's pretty easy.

You there's a command line utility. So you run cosine generate key pair. And in this case, I'm specifying that it go into my kubernetes cluster.

So this is going to generate a key that is going to be in my kubernetes cluster that's available for and this is the cluster that I use for building things so I can verify that an image was built using that cluster because the keys are only stored and only created on that cluster. So once I've got that key created I can then sign an image and upload it to a registry so I could do that manually on the command line if I have access to the keys. I also created for everybody's benefit a tool in the code fresh workflow template Library.

So if you're not familiar with this, let's just go look at it really quick. This is the code fresh argohub. Or this rather.

It's the codefresh hub for Argo as you can see and in here, I created this workflow template. So if you're using our go workflows to build images and you want to sign them you can just grab this workflow template off the shelf throw it into your workflow and you're good to go. io slash argohub.

So let's go back and once you've done that you can put it into your workflow template and you can see here an example of how that was done. You can see we we basically just pass in the cosine Keys. These are pulled from a secret and then in the template it just references which image needs to be signed and the tag needs to be signed and then we need to make sure that we have our Docker config mounted because this is going to allow us to push it to the registry that is there.

All right. So once we've done that. We have another tool that we're going to use called Opa now many people are familiar with Opa but they're not familiar with Opa gatekeeper.

So Opa is stands for open policy agent and it is a very popular general purpose security policy engine. So it gained a lot of popular popularity off of Istio and it's implementation with Envoy proxy because you can set policies for how Opa deals with those kinds of things. So for example, you can basically set firewall rules on the service level and things like that.

This is a graduated cncf project. It's very well baked. It has a ton of tooling it's it's very excellent.

Now, it's a newer implementation of opa is Opa gatekeeper. This is a kubernetes controller that sits on your kubernetes cluster and you can give it policy and it will check using in ingestion web injection webhooks to make sure that those policies are implemented so you can do things like say make sure That this image is signed before it's allowed onto the cluster. com slash open policyagent slash gatekeeper library and they include things like protecting specific namespaces preventing privilege containers blocking and allowing image repositories ensuring requests and limits are set limiting how many replicas there are blocking wild cards all kinds of stuff and you can actually write your own in a language called Rego, which I wasn't ever familiar with Rego is a little tricky to learn but because the library is so large.

I feel like Opa gatekeeper is is a great resource and a great one to use. So the way this is going to work. Is when we make changes Argos CD we didn't even talk.

I don't even introduce you to Argos CD. If you're not familiar with Argo City Argos CD is a tool that's part of the Argo project and rocd basically lets you set as a policy. This is an application.

This is where I wanted to deploy to this is where it's manifests are from so you define the source of truth from git and the destination in a cluster and then Argos CD will follow the policy. Make sure those are kept in sync. You can say if something changes in my actual State.

I want you to override it with my desired state from get and that's called Auto healing self healing. And then you can also have it do things like pruning. So when I delete an application something and get I wanted to automatically be deleted from my resources and you can basically set policies around that so we're gonna be using Argos CD as our gitops engine our get-offs operator if you will and what what we're gonna do is when we make a change our go CD is going to apply those manifest to the kubernetes cluster.

That's going to trigger an admission web hook for Opa gatekeeper to review policy and apply constraints. So there are two components to Opa gatekeeper. There are policy templates.

These are things like like I want to check namespaces. For some criteria and then a constraint is the actual policy itself. It says check against this policy template which checks namespaces and these are the values that I'm looking for.

And then once it's reviewed those things if it passes then Opa gatekeeper will pass it on to the scheduler and kubernetes will continue deploying it if it doesn't meet policy it's gonna be rejected. So I mentioned that Opa Library exists, which is a huge library of policy templates and these import are implemented each one as a crd. So as a custom resource definition and you can see that I actually think them using our CD.

So I just have a customization that references the OPA library and then Argo City takes that and renders these and deploys them into my into my cluster work at OPA gatekeeper to use So let's do some. Demos here. Let's use opiate gatekeeper to apply some.

Policies. So let's see if I remember how to do this demo. So I've got a couple of policies that I want to apply no floating tags only.

Allow me just actually before we do this we need to deploy it so I'm looking at my Cluster here and you can see I don't have anything deployed other than I've got Argo CD managing itself. I'm using a tool called argocd autopilot. Argo see the autopilot provides an opinionated straightforward way of setting up Argos CD in a self-managing way so that everything is recorded in git and it goes into get first and it's self managing and it's awesome.

So let me let me bootstrap my cluster. And when I do this bootstrap of all my components by default in my repo, I actually had it as part of the normal bootstrap, but for the demo I wanted to bootstrap it for you. I don't know because I want to be special I guess so what I'm going to do is use Argos CD autopilot, and I've already passed in a git token and a git repo in my environment here.

So that what this is basically going to do is it's just going to make git commits for me. So let's let's actually look really quick at the Repository. So you can see this is my repository web everything defined and I have this folder called AppSec which is currently empty anything in my application folder is going to be synced to is is going to generate applications into Argos CD.

And what I've done is I've set up an application. I'm gonna be set up an application called source that is going to pull in the manifests the application configs from that repo. So I'll show you what this looks like.

It'll be more it'll be more obvious. So you app create and source? And I'm going to specify the app is at this.

Source folder I just showed you under the app configs directory. So this is where I put all my application configurations and specifically the ones that I want deployed onto OSS secure which is the name of the cluster. I'm operating on and the project is also secure.

So I do this this is going to just create a commit for me. And that commit is going to generate this. I configure Json which is going to be consumed by an application set into Argos CD.

So well, oh shoot. It's happening too fast. So you can see this is now triggering a bootstrap that's happening where there are a bunch of applications that are getting deployed and this includes my my autopilot.

Sorry. This includes my Opa Library. Let's look on this page here.

My Opa constraints my Opa Library. I've got my cosine provider. I've got gatekeeper running.

All this stuff is deployed and up and running and then you can see it's also deployed some of the stuff for my application. So I have a demo except app and I have a demo reject app that are being applied Okay, so we've got everything applied on our cluster and now we've got policies that we can actually test against so let's go back and look at this demo. and what we'll do is we're gonna go into.

whoa, whoa Oh, yeah, it's in here. Okay, so I've got my Opa GateKeeper. yamo.

Let's go into the examples folder here really quick. So I've got an accept Argo CD application. I've got a pod not allowed and I've got to reject not Argo CD.

So what I'm going to do here is Let's make sure that I'm using. All right. Make sure I'm using the right.

Cluster. Yep. I'm pointing at my OSS secure repo that has been generously been provided by our partners at AWS and by provided I mean they let us pay for it and demo it.

So what I'm going to do is in my example repo here, I've got this pod not allowed. So I'm going to do is I created a couple of namespaces in here and I made one that I'm going to apply security policies onto now, I could apply this on to all of them, but I'm going to be doing the secure namespace. So I'm going to try to deploy this pod not allowed dot yaml.

So I'm going to do K apply - F pod not allowed. In the namespace secure and I'm going to just do a dry run on the client here. And when I do that you can see it was created.

Oh shoot was my policy not applied. Well remember policy application happens on the cluster so I can't do a dry run client. I need to do a dry run server.

So if I do a dry run server, you'll see that this one actually violates policy. So this says that my gatekeeper denied the request for this to be ingested because this image is using a latest tag. I need to specify a specific version.

So I have a policy that set to reject if anybody tries to deploy something using a latest tag because that's a floating tag that would violate my get-offs policies our violent, I guess principles. So I'm not gonna allow people to do that so you can see that's pretty straightforward. It works really well when we do it server side.

It restricts the resource when we do it client-side, it doesn't because it's not aware of the injection web hook. That means that if I were to actually try to apply this manifest get rejected. It wouldn't allow it to deploy.

So let's talk about cosine with ci/cd. Now this is the this is the big demo. So now that you understand how Opa policy Gatekeepers working.

It's enforcing policies. What we want to do is go through a process and this is what the entire process would look like and we can trigger all of this. So starting with we've got two repositories over here.

We've got to get Ops repo. That's the one I just showed you and then we have an app repo the app repo is where we Define our application. So if we go back and look at this this Repository Let's see this repository just contains what is needed to run everything on my cluster.

So all of my if I just ran a tree on this really quick. Let's just do that. Let's go over here.

So you can see oh, there's actually quite a bit in here. So maybe maybe this isn't the best way to display this. But the way that it is organized follows the principles that are put out on repo organization code fresh has a great blog post on this.

and basically we get our all of the automatic generated AppSec are in AppSec and then we have our source repo that has two different sets app configs and manifest all of our manifests are in there anyway, so then we have our Then we have our application repo which is where these things are actually being generated. I'm using this color-coded app, and basically whenever I cut a release I want to trigger. The creation of my resources and sign them.

So if we look at this process we trigger ci/cd. This is going to run a build in code fresh using our go workflows in the code fresh platform and it's going to sign the image using cosine using that workflow template. I already showed you and then it's gonna push it to the image registry.

Once it's in the image rest registry. It's available to be deployed to the next step is I add the release to my gitops repo. Now this could be done automatically here.

I've actually done it manually but there's a great tool called image updater that would write this to get for me once I update the release and get out and I get off repo this will sync it using our go CD and first it creates the applications in kubernetes the admission controller runs, it will accept it in our resources and then it will deploy those resources and report this link status back. Right if it's if it's allowed so what is that look like, let's do it live really quick. 01 secure and this kicked off a CI pipeline.

I can show you what that looks like really quick just proved you that we're doing it. Let's just go into really quick. I'm not gonna trigger a new release because I actually haven't made any code changes since the last time I ran this.

All right. Let me switch over here. Okay, and I'm gonna go into my delivery Pipelines and I'll pull up my release Here.

So you can see this is the last release I ran was this 101 secure let's just look at this workflow really quick so you can see what it did. The release was triggered by the ones I cut the release this triggered the build it built my Docker image it signed the image using cosine. There's nothing to display here because everything ran properly and then it ran some report information did some image enrichment which is just native into code fresh.

And of course all that was pushed into GitHub. All right. So the first part of the demo for that is complete the next part.

Was to trigger the release so let's actually look at that really quick. So to trigger the release if I go over to my get-offs repo and I look at my manifests for my demo except app. And I look at what's supposed to be deployed for OSS secure you can see about this customization here.

And this is specifying what image should be deployed. Now. 1 slash secure now if I were to change this to an insecure release like blue purple or yellow this would be rejected.

So if we go back and looking here really quick, you can see that we have a demo app that's been rejected. Let's find out why this was rejected really quick. And if we look at the events in here, we'll see that this one was failed because we try to deploy an image that was from a repository that is not in the accepted list.

So we verified that it's coming from the right repository and this one wasn't allowed in my demo except application. You can see that it's currently running looks like it's doing a canary. It has done a canary release and completed.

Let's go ahead and trigger a new one and let's let's send you can already see that it's deploying properly when we do a security image Let's do an unsecure image. We'll just change this to Blue. Go Blue, it's not secure low.

And we're just going to commit that directly for the sake of demo speed. So this will there's actually no webhook setup from the get repo here you could but it doesn't rely on one. So within you know 30 seconds or so, it'll pick up and see that it changes happened and it will trigger the the deployment to go.

So this approach is pretty awesome because it lets you do things like make sure that people can't deploy images that weren't built where you expected them to be built. 5 this does server side apply. So that means when they tried to create this Resource Group, we got the event right away.

Hey, this is not allowed in this case. This was blocked because it was checking for the image verification and there was no signature on this. So this was blocking was picked up now because I'm doing a canary release all of my service.

There's been no service disruption here. 5. We actually did client apply.

So that means that it would have applied it but then it would have found in the cluster. Now it still work you would actually just see a little bit more spinning on this. 5.

So let's go ahead and switch that back to a secure release. And let's do this secure tag that I've already set up here. Go back to being secure.

Let's commit that. We'll go back to Argos CD. And and now I'm just gonna hit Refresh on it to trigger the refreshingly and you can see now this this has been applied.

This new resource. Looks like it's been applied. Oh, wait is this oh, this is just the past one.

So it's still going to create it. It's just trigger synchronization really quick. Oh, well actually.

It's already deployed. It's this one sorry so that's not a great demos. Let's do since we're already deployed it.

Let's switch to the the other. Secure image that I have. Because it's not going to trigger the canary release switch to diff secure release because I already had it deployed before so it was just sitting in the canary release.

So it was like yeah, everything looks great now because because you already had a deployed so you're just saying that you want to go back to the one that we already have deployed. Everything's fine. So now when we sink it will actually trigger a new resource.

A new rollout to trigger with a new replica set and you can see these these pods rolled out just fine and this is gonna go through its Canary release. So this will be tested and it will move forward as it's ready. Okay, so we just demoed that so using opio with Argos CD a couple of things to mention here.

So I tend to think of in kubernetes. There's kind of two kinds of resources. These are these aren't official but I call some fragile.

These are resources that once they're started broken. They will always stay broken and a good example is if you're deploying Ingress before and Ingress provider, it will get the wrong information on it and it will never be fixed. And then I think there are resilient resources.

These are if failures occur that will eventually correct. So a great example of this is if you try to apply a resource with a custom resource, but that custom resource definition has not been applied it will fail but as soon as the crd is added that customer resource will deploy just fine and that will just take place automatically based on kubernetes retries. It might take a long time maybe should have better policy around it, but it'll work.

So there are tools within our go CD one is called sync waves. This allows you to set the order in which resources are applied. And the other is retry retry basically, lets us set a policy for when and how often Argo CD should retry applying a resource So within our group Uh because we're relying on admission controllers Mission web hooks through open policy agent.

That means every resource is potentially fragile when it comes to applying security policy. Let me explain that differently. If I'm trying to apply a security policy to a new resource.

I have to make sure that security policy exists on the cluster before I try to deploy the new resource, right? So this is most important when you're doing a bootstrap and I'll show you how I did it on here if you're using so, oh, well just one other thing use retry on policies because the policies themselves. Are resilient they only require that the templates exist for the policies.

So we talked about those two things that were on the cluster and we also want to make sure that the policies are applied before any other applications. So to do this we're going to need to monitor app sync status and this does work with multiple providers. 5 we'll work.

Okay, you'll just get different kinds of Errors when policies are enforced. So to do sync waves, I recommend following this blog written by Christian. You can follow him on Twitter at Christian 814.

But this blog Argos CD application dependencies allows us to set the order in which applications are synced. So what I've done is And we bootstrap this earlier. I have a app of AppSec pattern in Argos CD.

Basically, this is an application that can that references to additional applications and one I called system requirements and what I called user space everything in system requirements has to be deployed before anything in the user space because system requirements includes all of my policies. 8. We removed this functionality and made an optional but basically we can do is we can say that when we're deploying an app of AppSec by default.

If this one failed it would still try to apply this one. So we want to prevent that. So what we can do is basically we enable health checks on our CD applications and we basically say this is a blocking operation if this if system requirements isn't applied fully I don't want users user space applied so you can enable that with this config map which is explained in the blog post.

I mentioned a second ago, if you're using it with our CDL pilot, here's an example of what that code would look like to inject that into your manifest and you can also find this in the OSS secure repo that I was showing earlier. Um, and then we can use sync ways for these AppSec. So sync waves specify the order so under my system requirements application.

I have sync wave 1 and under my user space. I have sync Wave 2 that means everything has to be applied in my system requirements through a group of applications which includes my policies and all that stuff before my user space stuff can be applied which is my demo applications. So I showed you what that looks like.

Yeah, we just did this. so we don't need to do that again. Let's move on to the next thing.

So another thing you can do is you can prevent resources from being created that we're not created using Argo CD. So this is a policy that I created inside of gatekeeper and the policy template exists out there. It's It's referenced in here.

And then all I do is basically I say check for these annotations and The annotation I'm looking for is the tracking ID that is set by Argo CD. Now you need to enable this level of resource tracking using Argo CD here. I set it on service and pod.

This is actually incorrect. It should be on deployment and replica set and things like that because the pod doesn't actually set a tracking ID. But by doing this I can make sure that nobody deploys an application outside of our go CD.

So since I already demoed what it looks like when you apply resource, and it doesn't work, that's pretty Prior to see it again. But there you go bonus tip. Kaiverno.

We haven't talked that much about kiberno. Kaiborno is a competitor to Opa gatekeeper and it has some additional things that not for this demo. I actually used Opa gatekeeper and kaiborno.

I actually have chivierano checking my image signing but then I have gatekeeper checking my other policies you can use both you could use just one. I just found that I wanted to do a demo that included both so I included both of them and they work fine together now Kai Varner does work a little bit differently because kyverno actually mutates resources when it applies calls policy. And this means that if you don't do something Argo CD will get in a fight with kaiborno.

Our go CD says apply. These manifests says looks great. I've added these information Argo CD says where did that information come from?

That's not defined and get over right that information and remove it says Hey something just changed. So I need to add my policy information up here or go CD says where that information. Come from and so on and so forth so we can actually fix this by doing a couple of things.

Basically, we can add an ignore differences section that and set a cluster policy that ignores all of the components that are auto-generated by kiberno. So those will not trigger resynchronization. So you can do that here.

The instructions are in the blog post that are linked at the bottom of the slide and the slides are course going to be provided at the end. So what are we accomplish? Well, we verified our software supply chain, right?

We made sure the images were built in the place that they were built. We made sure that was enforced using our open policy agent and caberno and Argo CD. We implemented it all with Argos CD to give us a much better enforced security policy.

People can't get around these things. Now, it's just default. It's gonna work this way and it's gonna reject you.

It's gonna reject your release. If you try to do something outside of process, if you try to deploy something not using our ghost CD you try to play something manually we can actually reject that and we can reject any kind of resource that people were trying to deploy that way and we did it with a hundred percent get off. Everything was in get and so you Can follow along if you want to go and implement this yourself you can take my repository that we just looked at this OSS secure repository.

You can bootstrap this onto your own cluster. Just name it OSS secure and you can bootstrap everything that I did right here and it will all work exactly the same tons of resources that I use for this to use the project repo artist CDL Pi I mentioned you can check out all the rest of these great resources after the Talk and of course as always go get certified. I this slide says over 10K students.

It's out of date. io get you to the same place and we have gaps fundamentals and Argos CD at scale are both available with that. I would take questions, but I think we're out of time.

So hit me up on Twitter at today was awesome. I would love to hear from you. Also go check out code fresh.

We have an amazing software delivery platform that takes Advantage of all the stuff and makes it easier especially to do it scale especially to do within a large organization. And with that I'm going to sign off. Thank you so much for coming.