Detection and Response Against Kubernetes-Targeted Attacks | SecOps Vision 2024

Hi, my name is Story Tweedy Eights. I'm the VP of marketing at kss o and today I'm joined by my colleague Jack Zarris. And we're going to be discussing, uh, detection in response against, uh, these attacks that have happened in 2023 that were specifically targeted, uh, toward Kubernetes.

And so, before, um, we get started here, I'll just invite Jack to give a brief introduction of himself, and then we'll get started. Hi, I am Jack Zarris, director of, uh, sales engineering for Koc. Been in the, uh, security realm for, um, geez, over 20 years.

And, uh, go ahead. I'll, I'll be presenting a little bit later, uh, after story's done. Awesome.

All right. So how we're gonna attack this today is, uh, you can see the four attacks, uh, that happened in 2023. I mean, there's, there's more that happened, but these are the four we're gonna discuss in detail today.

Um, and we're gonna go through each of them in terms of the kill chain. Uh, we'll talk about how you can potentially mitigate your exposure to these attacks if you're running Kubernetes. Um, and then of course, the, the main, uh, goal is to come away with some takeaways for how to do detection and response, uh, in a Kubernetes environment.

And then, uh, Jack's gonna come in and do a demonstration so you can actually see, uh, part of, of what we're talking about live. So with that, um, let's go ahead and get started with dro. Uh, DRO is a cryptocurrency miner.

Um, this particular attack was discovered by researchers at CrowdStrike. And, um, the goal was strictly crypto, uh, cryptocurrency mining. And this is a relat, it's a good attack to start with because it's, uh, relatively simple.

Um, so what they did to start out with was, uh, the attackers to gain initial access. They scanned for APIs, uh, Kubernetes APIs and authentication was set in these APIs to a, uh, setting where anybody could get anonymous access to it. So to really understand, um, what I'm referring to here, um, when we make a request to the Kubernetes APIs server, there are three kind of checks, let's say, that, uh, happen.

The first one is typically happening on the cloud IAM side. So it's just asking, um, is this person, uh, valid? Uh, it's, and it's not even a person.

It could be a person or a machine. And then we're talking about authorization. So this gets into r back, uh, in Kubernetes, if you are familiar with that.

And then, so if the, uh, person or machine is valid, then the question is, what can this person or machine do? And then there's a mission control which says, okay, is what this person or machine wanting to do acceptable and can it happen? So if I'm talking in the dro, the case of dro, um, around having access to the Kubernetes API, what we're talking about is really kind of that first, uh, uh, piece.

Um, is this user or machine valid? The piece that I just showed Now, it wasn't really stated in, um, the research necessarily, but you could see from the previous diagram that in order for that access point to work, you have to also have an associated rback control that says yes. Um, you can then get access, uh, if the API is open in this way to, to be able to create pods in the cluster.

So once these conditions are met, RAC and access, the attacker will deploy, uh, what's called a Damon set. And what that does is it allows them to put a pod on basically any pod, uh, in the cluster. And they use malicious images to, uh, to deploy the Damon set.

And then those, uh, pods would spin up the cryptocurrency miners. So relatively simple access point through an open API, uh, then deploys pods on the cluster with that access and spins up, uh, cryptocurrency mining. So in this particular attack, uh, in terms of of the takeaways, what we can, um, think about is, first of all, whether you know which, which point of the attack is actually unlikely.

So that actual entry point, the one that I described, is relatively difficult, uh, and, and not super likely to happen because Kubernetes defaults to the opposite of what that open API setting would be. And even if we're using Kubernetes in a cloud environment, um, the cloud environments also like E-K-S-A-K-S-G-K-E, they also don't allow for anonymous authentication. So somebody would've had to kind of like manually, like in a honeypot, for example, uh, misconfigure that API to allow the access.

Um, so really actually the most concerning part of this attack is the creation of the malicious pods once the attacker got in. Because in a Kubernetes environment, um, if you have that kind of access, your blast radius is, is basically huge. Uh, you can move laterally throughout the cluster.

Now, in Dro, the attacker did not move laterally, but you'll see later that other attackers do. So in this attack, the most important protective measure was, uh, preventing the creation of malicious workloads through the Damon sets. And you can do this, uh, by using an admission controller to stop workloads containing images from sources that are not trusted.

Um, the other thing that you can do is your observability tooling is generally always going to show a cryptocurrency minor because of the spike in traffic. Um, and then the other last recommendation for dro, um, if you are only able to see the, uh, pods that are created as well as activity, uh, in the API in like a polling interval perspective, if the attacker's able to get in and out super quickly as they can do in dro, um, you're not, you're not gonna see it if you're using those polling intervals to, to view the activity. So what you, you need to make sure that you're looking at this stuff, uh, in real time.

So moving on to, um, Minero, we're kind of going, uh, in a graduated way to, from most simple to most complex as we go here. So, um, Minero was relatively similar to DRO in that it used, uh, the exact same entry point for the attack. So, uh, open API has an associated RAC policy that allows the API to, um, to, um, or that allows access through the API.

In this case, what the attackers also did was, instead of using malicious images to deploy their own pods, the attackers deployed their, using the access they had gained, they deployed their own privileged pod. So, um, that gave them the capability further than in DRO to mount the host directory, which means they can escape the container. And through that, um, excessive privilege, they downloaded a root kit to try to obfuscate their behavior.

Um, and in the end, of course, uh, installed the custom mining. So, um, in the previous attack, no, no container escape in this attack, container escape is happening, which is, uh, very concerning. So differently then how we would look at this attack versus the last one, we're gonna have to set up some other admission control policies around stopping the deployment of privileged pods and host mounted paths.

Um, so that's a bit different than the kind of admission control that we would have set previously. Um, and but very similar, the entry point is similar. So we need, uh, the real time KSPM to be able to detect that, but also to detect this privilege container and host path mount behavior in real time.

'cause if the attacker's only doing it at a short period of time, um, we won't be able to see it if our polling interval takes hours. And again, um, observability is going to catch crypto mining activity. But also in this particular case, um, you're going to be able to catch, uh, crypto mining, uh, activity potentially with runtime.

Now it's worth mentioning that privileged pods and, um, well, yeah, privileged pods. So the Oasp Kubernetes Oasp top 10 list, this is one of like the classic Kubernetes security. You don't want this to happen.

And in terms of this attack, it's relatively, um, it's not super likely that privileged pods are going to be, uh, open by default. 13 and above, um, the default pod security policy can be privileged. And so you have to switch that privileged, uh, uh, policy manually, which makes it a little bit less likely.

You know, any added step, any added manual step is gonna make it less likely that a security measure is put in place. So this potentially increases the chances of, uh, exposure in, in this case. So let's move on then to our most complicated attack, uh, which is Scarlet Teal.

Uh, cystic researchers discovered Scarlet Teal. This particular actor has been around since 2022, but, um, only really started, uh, taking form in terms of targeting Kubernetes in 2023. So in this attack, uh, the attacker first looked at, uh, exploiting a vulnerable Jupiter Notebook web application that was hosted in Kubernetes.

And once inside, uh, the cluster through that exploitation, they tried to access AWS credentials. And, um, they were, and the, you know, they did this through, through a lot of different potential methods. Um, and they, they did get access to the credentials.

Um, and what they did with those was they used two open source projects. One was, uh, Paku, which is for AWS discovery. The other one is tis, which is, uh, for Kubernetes discovery and even pen testing to find, um, further privileges and ways that they could, um, compromise Kubernetes and AWS.

And their objective was to access, uh, some s three buckets and do so in a way that nobody could, could see them. But then their objective was also for, uh, crypto mining as, uh, as most of these are. So if we try to figure out, uh, what are the potential learnings from this particular, uh, attack?

So first of all, the entry point differently from Monero and Dro, in this case, the entry point is very, very likely to, to be an issue because there are multiple, multiple, uh, vulnerabilities and web applications. Um, there are new ones coming out all the time, so it's likely that this will happen, and we have to kind of start thinking as the attacker does in terms of Kubernetes in the cloud, just being a fluid environment, right? Not kind of one or the other.

So, vulnerable web app very easily flows into the AWS space. Um, and now the specific way that they did that movement is through an EKS specific. Uh, it's, it's not exactly a vulnerability, but it's, um, an, uh, a way that attackers could, um, easily find a weakness in the environment.

So basically, the instance metadata version used for, um, the roles associated, like the cloud IAM roles, um, was version one, not version two. And with version one, there's a way that attackers can get in. So with that, what we need to learn from this and understand is that there are specific guardrails to look for and to implement for, um, each of the different cloud environments, in this case, EKS specific instance, metadata service, version one.

Um, the other thing that we need to look out for in, uh, this particular, uh, attack is this. So, you know, they use those open source projects to really bounce laterally and find whatever other potential areas of weakness in the environment. So how to detect the lingering persistence and lateral movement in Kubernetes.

And the way you do that is by understanding, um, not just the permissions that are given and potential over permissions, um, because these attackers, they were able to create completely legitimate permissions and then use those permissions to, to carry out their attack. So if we were just looking at over permissions, we would never see that. We have to look at the actual usage of valid permissions too.

Um, and then of course, to detect the, the persistence in lateral movement, again, this point of real time. So being able to see misconfigurations that the attackers could use in real time, because attackers will move very, very fast. Um, and then, uh, the last point to make on Scarlet Teal is really just the CVEs.

So the web application CVEs, how do you find, uh, from a broad perspective, um, the, the CVEs, not just before production, but in like the CVEs that are present in your running environment. Not all scanners will take that particular approach. So moving on here to the last one, the easiest one before, we'll go onto the Superfund part, which is the demo, um, rrb back buster.

So this attack as from this year, was found by researchers at Aqua Security. Um, it's similar to Dro and Monero in a few ways. Um, so basically the attacker, again looked for an open a misconfigured, API server, um, that would allow unauthenticated requests with one tweak that they, they were looking for a misconfigured server that would allow unauthenticated requests from users with privileges.

That's like a double whammy, uh, basically that frankly is quite, um, unrealistic to happen in, uh, in kind of reality. Um, and we'll get to that in, in a minute. Um, but that's how they got in.

So they had privileged access, they basically had access to the entire cluster to begin with. They used that privileged access to list secrets, discover the cube system, namespace, uh, even check for a few other attackers to delete them. Um, and in the end, they tried to create persistence with RAC.

So they bound a new cluster role with a new service account, uh, giving admin privilege to the service account in the namespace, which was them that they could then use to do crypto money. So getting back to what I was talking about before in terms of just how you know, exposed, could you potentially be to this, the entry point is, is, is simply, it's, it's relatively unlikely, but, um, it's still, uh, important on the RAC piece of this attack to understand the bindings. So the bindings of RAC that actually give, um, uh, certain privileges to service accounts.

So that's like the machine element. It could give, uh, the binding of a policy could also give, um, privileged access to a user. So RA auditing, you wanna see who does have over permissions in your environment.

Um, and then again, you can also stop workloads containing images like we did in the very first attack, uh, dro, um, from images, images, um, that don't come from trusted sources. So to summarize here, the top, uh, elements of detection and response in a Kubernetes environment. Um, first of all, observability tooling is gonna catch crypto mining, uh, in most cases.

Um, real time being able to see in real time the creation of the malicious pods, uh, the creation of a host mounted path, things like that. Um, real time's very important. But then to stop the creation, admission control, right?

That's gonna be, uh, um, the key response mechanism there. Being able to search across a broad environment for CVEs and web applications. Uh, and you can do this with either a searchable sbo m like the K koc has, or just turnkey detection across the board, but again, in your running environment, not just before.

'cause that can sometimes create a lot of noise. You have to do it for your actual running environment. And then how to uncover hidden roll bindings, uh, doing this with our back auditing and understanding, um, the permissions that are actually being used, not just the over, uh, over permissions in your environment.

Um, so with that, we are going to, uh, get into the demo part of this. And Jack's gonna talk about basically what polling intervals we'll miss, and I'll let him take over from here. So polling intervals.

So most of the tools out there right now, uh, with the exception of the ones that are looking at runtime, they're typically gathering, uh, information, uh, you know, once a day, once every six hours, something like that. Um, so, you know, if I was to roll out a deployment with a misconfiguration in it, uh, rolls out one 20,000 pods with a cv, critical CV in it. It's exposed to the public, it's over permissioned.

Uh, this is something that, uh, a lot of these tools wouldn't see, uh, for hours, uh, maybe a good 24 hours. Uh, and if that misconfiguration is pushed out, um, let's just say for an hour, and it could be exploited five minutes, it could be out there five minutes. And if there's, if it's a large company, uh, it's probably getting probed constantly.

So if there's a vulnerability found, it could be exploited in just a few seconds. And to, to illustrate that point, uh, let me share my screen here, one second here. So here, just a few minutes ago, three minutes ago, it's, uh, 10 33 my time, uh, 10 30, I just rolled this out three minutes ago.

I pushed out a, uh, a vulnerable, um, application, a vulnerable deployment, um, that has a, a, a workload that has a cluster role with that access to secrets. Uh, it's also allows privilege escalation, and it has a public exposure. And, uh, I knew about this within about a minute of, uh, of rolling it out.

So I go ahead and, uh, say, click on this one container where that allows privilege escalation that is exposed. I can see that threat vector here that's showing you the exposure through this ingress. Um, through, uh, this is the deployment here.

It allows privilege escalation. If I want to take a look at that. I can see the, the, the YAML here.

Indeed, it does allow privilege escalation. It's given me the remediation advice. This comes out within a, a few seconds of that being pushed out.

The, uh, the link to the ingress happens in about a minute or two, and I get that threat factor. Uh, but it's warning me that, you know, this, this is exploitable. It allows privilege escalation, it's exposed to the public.

And I can go in here very quickly and see, oh, are there other misconfigurations other than this one? And indeed, uh, there are quite a few here. Um, the image tag's not set the latest memory and CPU uh, limits are not set.

Not all these are high, but it shows you all the misconfigurations, even if it's not considered a threat vector, that means it could be, uh, exploited. I can see all the misconfigurations here. So forensically, uh, when I see this, I wanna see are there other issues with this that I don't know about?

So I can see all those misconfigurations. Um, I can see are there any runtime alerts in here? And indeed, yes.

Uh, to the point here, this was exposed and within a few seconds or a few minutes, there was an attacker in here. Because this site is being probed constantly, uh, I am probing this, uh, this, this system constantly. And as soon as a vulnerability comes up, a script automatically starts exploiting it.

So it's, you know, if I, uh, go through the different pages of this, what this is doing is it's, it's exploiting a vulnerable web application DVWA, if you're familiar with that. And it's setting up a reverse shell that pulls down a script that installs curl, then installs QCTL checks to see what permissions it has and immediately starts extracting secrets. So I'll go ahead and run it again here real quick.

It takes about eight seconds to run, and it extracts secrets. It pushes those secrets out to an attacker on the internet. And just there, it's done.

Now, uh, just for good measure, and I'll refresh this screen and we'll see the additional runtime alerts coming in here as I read it. So seconds ago, uh, extracts the secrets here, uh, pushes them up to, um, an attacker out on the internet, uh, removes curl from the, from there in case somebody goes in and says later and says, what's curl doing here? And just for good measure, uh, it tries to apply a pa a bad pod, um, that establishes a persistent reverse shell if this hat pod happens to go away.

So all this happens in seconds. And if I go and fix this issue, like, oops, you know, I pushed out a workload that's privileged, it's not supposed to be there, um, it's only up for a few minutes, I go fix it. Tools that scan for misconfigurations will completely miss that this ever existed at all.

You need to be hooked into the runtime event stream continuously to be able to see and notify teams immediately when this threat vector exists, when this is introduced in the environment, not if it's introduced and it happens to be there hours later, then I'll be informed. I need to know in real time when something is pushed out that's exploitable because it can be exploited immediately. Um, um, now obviously, you know, we have admission control or maybe not.

So obviously we have an initial controller where, um, you know, if I try to push out a pod that establishes a reverse shell, I can set that to block. I purposely allow this stuff in here, but that can be set to block, and then I can get an additional alert to say, not only do I see a threat vector in, um, in a vulnerable pod that has CBEs, um, it's exploitable, I see runtime, but oh, and I blocked a, uh, a bad pod that has a crypto miner or setting up a reverse shell, um, from being deployed from within this pod. Uh, so that's easy to set up as well to block.

And I would get notified via Slack as well. This isn't supposed to be a comprehensive, uh, walkthrough of the, the system, but the last thing I'll show here is, you know, all the different threat vectors that in this, that exist in this environment, some with the same app. This one allows privileged escalation.

Allhouse has a surface account with excessive permissions, critical vulnerability that's exposed. Now, I have thousands of critical vulnerabilities in this environment. I have hundreds of misconfigurations, uh, but I have a very short list of threat vectors here, so that we're not bothering with a noise.

We're only letting you know when there is a cross section of issues across our back. Misconfigurations, um, uh, misconfigurations, um, um, uh, public exposure, things like that, that need your attention to tell you. This isn't just a misconfiguration, this is something that's exploitable and you need to know about it.

Now, uh, I'll stop here and I'll hand this back over. All right, so that was, uh, Kubernetes detection and response in, uh, very short amount of time. And thanks so much for, for joining us.

If, uh, you have any questions or, or, uh, comments about these attacks or how to do this with ks o, uh, please feel free to reach out to myself or Jack. Thank you, Jack, and thanks everybody.