Liz Rice – The State of Kubernetes Security

Hi, I'm Liz Rice. I run the opensource engineering team at Aqua Security, where we have a platform for helping enterprises secure their cloud native deployments. And you might also have come across me through my work with the Cloud Native Computing Foundation, where I'm currently chair of the Technical Oversight Committee.

And I'm going to be talking a little bit about what I consider the state of Kubernetes Security is today, I've written a book. It's actually a couple of years ago that Michael Hausenblas and I wrote a book on Kubernetes security. And then this year, I have a new book called Container Security.

At the end of this presentation, I'll share a link where you can get copies of both of these books. You can download them in electronic form from the Internet, or you can go to your favorite local bookstore and buy a copy of container security. You probably have to order it.

I doubt it's on all the shelves, but I hope if you buy a copy that you'll really enjoy that book. The idea of the container security book is to teach you really what's happening with containers and what the consequences are of using containers, how you can think about them in security terms and what things you can do to improve the security of running containers. And security is a very important topic.

And if you're considering moving to the cloud, moving to cloud native, security's probably one of the top issues that you're considering as a challenge. These are the results from this year's CNCF survey. And as you can see, security is the second most mentioned challenge that people are worried about when they're thinking about moving to containers, second only to the cultural challenges of moving your organization to DevOps or DevSecOps organization.

And if we think about why security would seem like a challenge, particularly for a security professional who's used to a traditional deployment, it's not really that surprising that it raises some concerns instead of having a set of machines that are well known and that are managed individually. We now have a cluster of machines. Software could be running anywhere within those machines.

Quite typically, we might see hundreds of different micro services these days running in thousands of containers. And the lifetime of each of those containers is really small, according to some research by Datadog, the lifespan of your average container running in an orchestrated system is less than one day. So if you're a traditional security professional or other security professional who's used to a secure traditional deployment, it's no wonder that you would have concerns about moving to this new world and this new way of doing things.

Plus, if we're moving to Kubernetes there are a whole new set of attack vectors that you might need to be thinking about, this diagram is adapted from the Kubernetes security book that I mentioned earlier. And as you can see, there's quite a few different routes that an attacker might choose to use if they wanted to compromise your deployment or try to get hold of your data or do some kind of malicious action to your deployment. There are a lot of arrows on this diagram.

It's complex. Kubernetes is complex. And unfortunately, as Bruce Schneier said, complexity is the worst enemy of security.

And he said that back in 1999. But it's still true today. And there's no getting away from the fact that a distributed orchestrating system is going to have complexity that's inevitable.

So how can we address that and how can we tackle the problems of security and try to make this more manageable for individuals who are going to deal with the security of a cloud native deployment? Well, one approach is to have better defaults, and as we'll talk about shortly, the default settings for Kubernetes have improved greatly in terms of giving you better secure default settings. Another thing that we can do is bring automation into the mix.

If you can automate things, you can remove the risk of human error. And then the third thing that you can do to address complexity and this could come across as slightly facetious, but it is true, is you can get somebody else to do it. And in particular, when I'm talking about cloud native security, I could be talking about using managed services.

If you're using managed services, you can have somebody else take responsibility for some of the things that you no longer have to deal with. This kind of shared responsibility model, I've used the diagram from AWS here, but it's common through all the public clouds where the manager of the service will take responsibility for the security of the cloud and for the pieces of infrastructure that they are managing for you. And then you as a customer need to take responsibility for everything that you're running in that cloud.

The more services that you ask someone else to manage, the less you have to worry about the security of those services. So you might be using virtual machines or even bare metal machines in a public cloud. And in that case, the physical security of those machines will be taken care of for you and the access to those machines.

The failure is all looked after for you. If you move up the stack and ask a cloud provider to give you a managed Kubernetes service, that means they can take responsibility for the control plane of that Kubernetes service. And that means they're going to take responsibility for the security of the controller.

So that takes care of quite a lot of the attack factors that we saw in the previous diagram so we can remove those from the picture, you're still going to have to take care of restricting access to machines and restricting access through the APIs. And, of course, you're going to be responsible for the applications that you're running. If you're not taking the managed services route, you're going to have to take control of the security of those control plane components as well, and there is guidance to help you with this.

So here I'm talking about what you can do to configure all these different components, particularly the Kubernetes components, to make them as secure as you can. And the guidance that I'm talking about comes in the form of Benchmark's the Center for Internet Security, publish a Kubernetes benchmark. And it has probably 200 pages of recommendations for how you should configure those Kubernetes components so that they are best practice from a security point of view.

And the good news is that compared to a couple of years ago, a lot of these recommendations are actually now the default settings and that's a big step forward. So if we think back to 2018, only a couple of years ago, there was this very widely reported attack on Tesla where essentially they had left their Kubernetes dashboard open to the Internet. Attackers use that to install cryptocurrency miners and essentially freeload on Tesla resources.

Since then, things have come on a long way. Lots of things have improved. One obvious change is that the dashboard is no longer left open to the Internet by default.

In fact, by default, it's not even installed. You have to add that yourself if you want the dashboard. Anonymous access is now turned off by default, which is a good thing you no longer have to know to check to turn off anonymous access to the APIs.

And the access permissions over those APIs are very restricted by default. So as a user, if you try to use the API, if you're anonymous, if you don't give an identity for yourself, you're unlikely. And certainly with the default settings, you will get very little useful access.

And even if you identify yourself, unless access control has been set up for your identity, you'll have extremely limited access. Now, if you're thinking about how those access permissions are configured, this is now pretty much universally done by role based access control. In the past, there have been other ways of configuring permissions, but now role based access control is the way we recommend to do this in Kubernetes.

So. Role based access control is really saying how you're going to grant different uses or groups of users or pieces of software permissions to perform certain actions on certain resources. I'm sure we've all.

If we've been administrating a cluster, we would have had someone ask us for permission to do things in that cluster in this example. Jennifer says she needs to be able to create pods. She's doing the action of creating resources that are tight pods.

So in order to set them up, we need to be able to identify that it's Jennifer and we need to give Jennifer permission to create pods. The first part of that is authentication, identifying who the user is and unfortunately, maybe not. Unfortunately, it's part of the design that Kubernetes just doesn't have a first class resource representing users.

It expect users to be identified by some third party provider. So there are various different authentication plugins that you can use to integrate with and ID or HELDA app or whatever identity provider you're using in your organization. That authentication piece says which identity is making a request, so in our example, it's confirming that this is Jennifer who wants to create a pod, having identified who that user is, we then need to check whether or not that user has permission to perform that action on that resource and in role based access control.

This is quite complex. We have these two different abstractions. We have roles and role bindings.

So a role says if you have this role, you can perform a certain set of actions on a resource or group of resources. But it doesn't say anything about the identity of the user or the software that can take that role. It's a role binding that links a role to that user or software identity.

This having these two different types of abstraction, the roles and role bindings, and there are also cluster wide version, cluster cluster of bindings, having these two separate abstractions allows for really granular control of who has access to do what two different types of resources in your cluster. But it's confusing. It's pretty hard to make sure that you've got the right set of rules and the right set of role binding set up.

As part of trying to address that, my team and I wrote a key control plugin called Who Can? And the idea of this plug in, which you can install with the plug in manager called crew is it can tell you all of the users, all of the identities who have permission to perform a certain action, certain verb on a certain resource type. So in our example, if we wanted to check who has access to create pods, we should see Jennifer in that list.

And as you can see from the examples in the help, there are lots of different ways that you can query the role based access control configuration, perhaps across just one namespace, all across all namespace. It's. So if we check who can create pods in my example here, Jennifer actually has two different ways that she's granted that permission.

There's a role binding called run pods, which is granting her that permission. But there's also a cluster role binding or create pods anywhere. And Jennifer is mentioned in that role in that cluster role binding as well.

So. You can use this kind of query to figure out who has access to the different types of resources across your cluster. I think this is indicative of the kind of thing we need to improve to make it much easier to manage role based access control and make it very clear who has access to do different to perform different actions within the cluster.

So role based access control is how we can limit access through the APIs. So the other type of attack vector that you're really very definitely going to have to be careful about takes on some actions around, is around securing your applications. So you're going to be running applications within Kubernetes that are going to be containerized applications.

And if they have vulnerabilities, those vulnerabilities can be exploited. And there are a lot of different steps that we can take to improve the security of containerized applications. Some of the best practices that you should consider and probably should be doing first on, I want to mention, is using dedicated build machines.

It is possible to run your builds on the same cluster as you're running your production workloads. It's possible to do that securely. But unless you really know what you're doing, I would recommend separating them and keeping your builds separate from your production workloads.

And the reason for this is that today most build systems, most CI/CD, pipelines are using privileges. In order to run docker build, they will typically have access to the dock socket. And that basically makes the the build machines the privileged.

So if they get compromised, whoever manages to compromise that machine has some privileged access. So keeping that separate from your workloads, your production workloads, it gives you that isolation. You want to keep the possibility of a compromise build machine away from your real data and your real applications.

It's very common to use a trusted set of base images, so the front line and all your docker files would be one of these trusted base images that has been approved perhaps by the security experts in your team who are confident about a particular version of base image, a particular distribution of Linux. You might want to specify a version of whatever programing language that you're using. You might want to use a particular python image or a particular go image or whatever language you're using.

Specifying those trusted based images means there's less variation across your container portfolio, if you like. Another thing that is highly recommended that you do within your docker file is specify a user, make the user that's running the identity, that that container is going to run under a non-root user by default. When you run a container, it is reached inside the container.

And in most systems today, that means it's root on the host. I think in a year or two years time, we may not be focusing on this so much because it's been a lot of progress being made in what's called rootless containers, where you can run root user inside the container, which isn't the same as root on the host. But for most of us running Kubernetes today, we're probably not using rootless containers.

So for for now, the best practice is to specify a user in the docker file and make sure that your container doesn't run as roots unless it really has to. Another very common recommendation and something that's very easy to build into your CI/CD Pipeline is scanning those container images for vulnerabilities. These could be vulnerabilities in the OS packages.

It could be vulnerabilities in the application language dependencies. So there are lots of different image scanners out there. I would recommend making sure you pick one that supports the particular distribution of Linux that you're using, or perhaps you even have Windows containers so that there are scanners that support windows as well.

For most of us is Linux. So look for the distribution support that you need and look for language support. If you're currently using Ruby, look for a scanner that can support looking for vulnerabilities in Ruby packages.

Once you've built your images, it's a good idea to store them in a private registry. It's just an extra layer of defense if your your images are coming from a registry that's open to the Internet. That's just one more risk that perhaps something will get compromised.

So what we see with most enterprises that we talk to is they're running a private registry or they're using the registry built into the public cloud so that it's not accessible to users outside your organization. So those are a few steps that just revolve around building your container image and at the end of that build process, storing it at the point where you going to deploy it? There are more best practices that you should really be doing.

One is to check the well, first of all, check that the image has been scanned and validate that the scan results didn't show any significant vulnerabilities and that that scan happened reasonably recently. For most of the organizations that we speak with, scanning happens on a daily basis. If you have an out-of-date scan, there's a risk that new vulnerabilities have been found in the code inside that image.

So you want to be able to risk images or at least check that the scan has been performed pretty recently. You don't necessarily want to run the scan at the point of deployment because scans can take a little while and even if it's just a few seconds and necessarily want to delay deployment, but you want to check that the scan results are recent. You may also want to check the configuration of that container, when to check for things like not configuring containers to run as privileged or check what volume mounts.

The container is set up. You probably don't want to be mounting the etc. directly from your host into a container unless you really, really need to.

So that kind of check is something you really want to automate. In the past, it's the sort of thing that you've been able to do and you still can do with security policies, but I'd say that if we're thinking about the current state of Kubernetes security, we're moving towards more and more people using open policy agent for this kind of check. So open policy agent can provide you these kind of safety rails to check what's happening before a resource is deployed.

There are lots and lots of different questions that you can essentially ask with open, open policy agents. For example, if you've been given a pod specification and that includes container with a volume mount from the host, is that volume mount acceptable? Should the application be allowed to run as root?

If it's configured to do so occasionally, you might have good reasons why you want to run. An application is root, but in most cases you don't. You might be checking things about other kinds of resources, so, for example, was it OK that we gave Jennifer role binding that allowed her to create pods might want to have policy rules defining what role bindings are permitted in your organization.

And there's an endless list of these kind of policies that you might want to apply in your cluster. OPA lets you define those policies. Ask these questions using language called Rego and you can see open policy agent enforcement points, for example, this gatekeeper, which can check as an admission controller before the API server is allowed to create a resource that goes through admission control.

And we can check OPA can check whether the policies or any policies that apply to that request on that. We can also see open policy agent being used in lots of other situations around the cloud native world. And I think this is an exciting part of the puzzle from a security point of view.

And it's a good example of how automation is really important. Scott Hanselman said the most powerful tool we have as developers is automation. I think particularly in a cloud, native world automation is also the most powerful thing.

We have to help us with security. Now, I want to show you an example of a new piece of automation that we're working on in my team. It's a project called Star Board, and the motivation here is that today we have lots of security tools, lots of things that can automate security for us.

I've mentioned scanning images for vulnerabilities. I've mentioned the CIS benchmarks that were tools like our own Kube bench that can be used to check whether or not your cluster is configured. According to the benchmark I talked about checking the configuration of your pods.

You might use a open policy agent to police that. There are lots of other tools that will scan your configuration and report on any potential issues. Pen testing you, you might have a budget to employ a human pen tester, you can do a thorough job of checking whether there are any weaknesses in your cluster.

But if you don't have that kind of budget where you only have that budget occasionally, you might also want to consider using automated penetration testing tools. And all of these tools today create different types of output reports. They might be generating documents, they might be writing information into databases.

They might have different formats of Jason or text output, all kinds of different possibilities from different security tools that come from different vendors or different open source projects. And the person who's looking after those security tools or reading those reports may also be someone who is used to using the Kubernetes tools that access Kubernetes through the Kubernetes API. The idea of stubbled is that we bring those tools into the Kubernetes API so that you can get that security information using those familiar tools and that you can trigger different scan's you can trigger different types of security test and then read the reports out in the form of Kubernetes custom resources.

And the idea is that you could plug in different types of third party tool and generate these different custom resources from whatever tool you want to use in your classroom. Starboard is another one of these crew plug ins, and you should definitely check out crew, there's a list of many good, useful plug ins that enhance the crew control interface. So definitely check out crew.

And then when you're there, you can install starboard using crew that you can then access starboard through the key control tool. And as you can see from the available commands, I'm going to focus on, you can run in it, which creates those custom resources you can use find to manage security scans and trigger those security scans to be to be initiated. You can get those security reports once they've been triggered.

So let's see this in action. All right, I have a single node cluster here and I have let's check the deployment's. OK, and I have a deployment running, an enginex pod, and because I set this up earlier, I can show you the notes I need to get correctly.

I can show you the custom resource definitions that have been installed. And these were all actually set up using the starboard in its command. And if I wanted to trigger a scan on that deployment, I would do a starboard find vulnerabilities in the deployment.

Engine X, I'm not going to run that now because I already ran that earlier on, so let's quit out of that. But I should be able to do control get vulnerability reports and I'm going to use wide output. And this is the report that got generated, was running over an engine X image happen to be the tank one dot sixteen, which I happen to know is a bit of an old image and does have some vulnerabilities.

I can see that we use the Trivy scanner to generate this report and it created some of it, detected some vulnerabilities, didn't create them. It detected them. So I've been able to get this information about the vulnerability from the Kube control interface that we're all familiar with.

And if I were to use Starboard. Is an deployment engine X, we see a long list here of this is very detailed vulnerability information. So although that's potentially very useful, it's quite hard to read as a human being.

So the other thing I'd like to show you is a plug in into Octant so Octant is an open source dashboard for Kubernetes. And if I come and look at my Engine X deployment in here, we can say there's a vulnerability's tab here. This was created by having a star board plug in for Octant that we've written.

And if we come in here, we can see all the details about the vulnerabilities in that particular instance of Engine X that I'm running. So this is an idea that we've been working on, it's still very early stage, but really about bringing this security information right here next to the information about running workloads, making it accessible over the Kubernetes API. So if you want to check that out, you'll find it on GitHub under Aqua Security Starboard, and we'd love to hear your feedback and your thoughts on why we should go with that next.

OK, so getting back to the slides and thinking about attack vectors, I've discussed a lot of the arrows on this slide so far. There's one left that I haven't discussed and that is escapes from the container to the host. So there's some way of compromising a container to get access to the host machine that it's running on.

Now, there are ways that you can achieve this through bad configuration. I mentioned earlier checking your volume mount. If you mount the root directory, that's a really kind of crass way of achieving container escape, because if you mentioned the hosts directory into the container, the container code could do anything you liked on the host.

That would be pretty, pretty crass. The kind of containers you really need to worry about or perhaps not, don't lose sleep over it because they are rare, but it's the kind of vulnerability in container runtime code or perhaps in the Linux kernel that allows the containerized application to escape onto the host in an unexpected way. Now, these are rare.

They are not something that you should worry about too much, but they have happened. And there was a recent case just in the last few weeks, CV 2020 for the year and then 14386, and this was a privileged escalation, allowing containerized application, if it had the capability called cabinet rule, allows it to send raw network packets, that could be essentially a bug in the kernel, allowed it to gain extra privileges and escape onto the host. So one thing to take away from that is make sure you've upgraded your software to take the patch for that issue, to don't allow containers to have capabilities unless they really need them.

We could probably spend half an hour talking about capabilities and how you can restrict them. Cabinet rule is one that's rarely needed. It's it's traditionally kind of allowed because things like Paint need cabinet rule.

But most applications, unless they're running something like Paint, don't need it. I think this is another area that will improve on it is what going on in the container run times to and indeed on Linux to make it less likely that you actually need Cabinet rule and have a ping that doesn't need it. But for now, it's a good, good idea not to give capabilities that are not actually required by the application.

But the other really interesting thing about this particular vulnerability from the perspective of containers escapes is if you were running your container inside the gVisor, you are not subject to this escape. It was not going to affect you. And gVisoris one of a class of kind of sandboxing techniques that provides extra isolation between your containerized code and the host that it's running on.

And as far as I know, this was the first case where there was a clear advantage to using that sandbox because you could not have the compromise from this particular vulnerability if you were running gVisor. Now, the other thing to be aware of, though, is that as far as certainly as far as I'm aware. This was not exploited by bad actors.

The the issue was discovered and the vulnerability was fixed before it was abused, as far as I'm aware. And if you know differently, I would love to hear about it. I think this brings us to another important point that Kubernetes is software, you have to keep your software upgraded and it's complex.

We talked about that earlier. Complex software will, from time to time have vulnerabilities discovered in it. And you need to keep it up to date if you're using a managed Kubernetes offering that will be taken care of for you if you're running your own Kubernetes cluster, that you really need to stay on top of this.

There's a that will give you the security announcements from the Kubernetes security team. Whenever there's a new patch that you really should be applying, this will tell you about it. And you can say, you know, there have been a decent handful over the last few months of things that you probably did want to do to patch.

The security team is you know, it's well organized, they have a very clear process for reporting security issues that a researcher might find in communities and an initiative that the CNCF have taken and particularly sort of invested in over the last few years is investing in security audits, getting third party specialists to perform these security audits on CNCF graduated projects, including Kubernetes and but also the other graduated projects as well. So there's been a lot of research, a lot of work done to improve the security of communities through this audit and then addressing issues that that audit found. What does it mean in the real world?

What are attackers actually doing to compromise Kubernetes clusters. So at Aqua we have a research team. And one of the things that the research team does is investigate what attacks are going on.

And they primarily do this by standing up what's called a honeypot, which is a cluster that maybe has a particular bad configuration, or maybe it's just a regular cluster that they want to see what what attackers might try to do with that cluster. By far, the most common type of attack that we see is attempts to mine cryptocurrency. Recently, we've seen a lot of attacks on the supply chain effect.

Effectively infected contain images that are being distributed containing cryptocurrency, mining software, malware, if you like. How do we detect this, one of the tools that I think is indicative of a new school of dynamic testing is it's called dynamic threat analysis. And the idea of DTA dynamic threat analysis is that you take an image, you run it, you see how it behaves, and you compare that behavior to known bad behavior.

To take a very simple example, if if you saw an image and it attempts to connect to a known cryptocurrency mining operation, then that's probably a sign that that image is compromised. So DTI is another potential tool that you might want to add to your armory, particularly if you're concerned about using images that you you can't be completely certain of their provenance, might want to run them through a dynamic threat analysis first to check for malware. Perhaps you want to do this for your base.

Images that as part of the approval process in an enterprise where you're going to have an approved set of base images. So I think I've probably shown that there's a lot of different angles to security for Kubernetes cluster, and I don't think it would be right to expect security to be something that you can just kind of turn on. We can improve the defaults.

We can make it easier. We should do those things. We should provide automation.

But I don't think we can expect everything to be solved by one silver bullet. As Neil MacDonald said from from Gartner said, the cloud isn't one thing. So securing the cloud won't be one thing either.

He said that I think it was in 2011. He made this statement. And I think it is still just as true today.

If we think about the approaches that you might want to take for automating cloud native security and think about three different pillars, you need to secure the build, make sure that the images you build are to the best you can make them secure to run. You need to secure the infrastructure, whether that's your own infrastructure or taking advantage of managed services and just having to police the things that you're responsible for. And at runtime, you want to be able to check that the work clothes you're running are safe and behaving.

As you expected, and not showing signs of compromise, so all of these different types of tool are things that you might want to consider when you're assembling your. Environment, your security approach for your cloud native deployment's. com where you can get hold of those electronic copies for the price of your contact details.

Essentially, if you have questions, I hope that by the miracle of modern virtual conferences, you've probably already been chatting with me, I hope, and asking your questions. But I do love taking questions. So I'm sure I am online as I speak, hoping to hear from you with your questions.

Thank you very much.