Eddie Glenn, Venafi | DevOps Connect 2022

Hi everybody. I'm really excited to be here with you today. My name is Eddie Glenn, and I'm going to be talking about how to secure kubernetes around the number one kubernetes security threat that your teams may be experiencing.

So I'm going to talk about a couple of things today. If you're familiar with identify, you'll know that we're experts in machine identities. And if you aren't familiar with what machine identities are, I'm gonna it's been just a few minutes explaining that and we'll talk about how the supplies to kubernetes.

And then I want to leave you with some real practical tips on how you can work with your teams to get them to address some of the the threats that we're talking about today. Before I get started, I just wanted to tell you a little bit about myself and about benefai. So I've been in the business for about 30 years.

It started off writing safety critical software. I've been involved at devops for for decades Agile development cicd pipelines and and software security identify was founded about 22 years ago. We have around 500 employees and we have a very large ecosystem of partners that really enable our customers to be successful at being able to use our Solutions along with the the tools they already use and as we all know and incal Native infrastructure, there are lots of there's a huge ecosystem.

There are lots of tools that are being used. We're also extremely Innovative. We were the inventors of this of category of Technology machine identities and machine identity management.

5 million dollars a few years ago to fund small startups as well as also some large startups to make sure that their Solutions work within within our platform and it produces a result. That's extremely valuable for customers like you We also partner with leading executional institutions. And then this just is demonstrated by the number of customers that are they're using our solution.

You can see here that our top five list across the board and lots of different Industries. So let's talk about the machine identities. I know some of you might be familiar with what we mean by that and somebody not.

So let's start off thinking about a typical Network, you know, it could be a network within your organization or could be the internet, but the way identify looks at it is that there are two actors on the network. And this is all around authentication and that for people obviously you and me, you know, whoever's using that particular computer network. We're going to use usernames and passwords.

So, you know, that's something that you know obviously is no surprise to any of us. That's just how it's done. But what you might not be familiar with is that the other actor on a network are the machines on the network.

So these are machines that are communicating with other machines on the network and they need a way to authenticate themselves to those other machines. So it could be you know, one data server talking to another data server. It could be microservices within a kubernetes cluster talking to another microservice within a kubernetes cluster, but there are lots of machines and those need to be authenticated as well.

And this is an area where you can see by this data authentication around people's huge industry, you know 11 billion dollars spent annually. And we're just getting started and you know, when we talk to our customers and especially their csos they don't always think about a managing those those identities that machines use and a lot of it involves, you know, really techno Concepts like encryption dating encryption encryption. And this is one of the things where we're seeing more and more breaches and allergies that are related to identities around machines and it's really an important topic for us all to me thinking about So when we think about machines and machine identities, you know, this this is a pretty good picture that puts the people down, you know towards the left but then you see examples of all the machines.

So when we talk about a machine, we don't mean just a physical piece of Hardware. We you know could be a virtual machine. It could be something in Cloud native like a kubernetes cluster.

Virtual machines, you know and Amazon Amazon Services Google Cloud platform Etc. It could also be services and apis so, you know, this is a way that software talks to other pieces of software and there needs to be authentication between that Applications so an application be considered a machine it and the application of piece of software has a machine identity associated with it. And we think about you know machines, you know being this kind of big umbrella of kinds of things.

That's just not physical Hardware but also software and virtual virtual devices we look at okay, what do we use to protect those and authenticate those to help ensure that you know, we're really dealing with the entity that we think we're dealing with so For you know physical Hardware. We're all used to SSL TLS. As we expand into Cloud native we start thinking about mtls Mutual TLS that helps microservices communicates securely for decades.

We've our it administrators have used SSH keys and certificates. That's a way to allow, you know us to log into one machine without having to have a human username and password. There are mobile certificates their code signing key.

So there are lots of different kinds of machine identities. And so when benefi talks about machine identities, we're talking about all these different kinds of machine identities. But today we're in particular.

We're going to focus on just TLS and TLS. and it's just not benefitted that that's in this business, you know, if you look at Gartner, you know leading industry analysts. They they see this as a hot a hot technology segment where it's extremely important for people to to manage your machine identities.

And and if you have a gardener account, I really encourage you to take a look at this this report from and so this kind of leads me into you know, what we're going to really be spending time talking about today and that is How do we protect modern infrastructure, you know, so, you know with with classic infrastructure we're talking about bare metal machines, you know data servers that sort of thing maybe even virtual machines. But when we look at what's happening people are migrating away from that kind of classic infrastructure. They're moving more to Cloud native infrastructure.

And you know, one of the things that's kind of been a little disappointing for me. Is that over the past decade a lot of a lot of businesses out there have you go into great measures to protect those machine identities with their their classic infrastructure, you know, they don't want their fives to go down. They don't want data servers to go down.

They don't want to have outages for their customers but as we've transitioned and migrated towards Cloud native. We aren't being quite as careful because those Engineers as Cloud native platform teams and application teams. They tend to be the ones responsible for managing those machine identities and if they don't involve infosec and Enterprise, Security, you know, there are definitely gaps that coverage and it's led to some issues that we're going to cover in just a few minutes.

And you know, I just want to leave on the slide with that. Everyone is transitioning from this classic infrastructure of cloud native structure. And we can't leave security behind.

We've got to make sure that security and especially Enterprise security policies and enforcement is carried Beyond just classic infrastructure and that it is addressed towards Cloud native infrastructure. And doing this though is going to call some new challenges for all of us that are owns on Enterprise security teams, and I hope that some of these tips that I'll cover later will address this challenges. So shouldn't be a surprise to anyone the use of kubernetes is exploding.

This was a report done by the cloud native Computing foundation and his last last December. 6 million developers are currently using it and that's a 67% growth over the over just one year. So even you know, if you happen to be on an Enterprise security team and you're involved or connected to your death secops teams.

They're likely already using kubernetes. And so then the question becomes how are they what they're doing plugged into what corporate policy and infrastructure is around around security. So this growth has a cost that's associated with it.

And this is a really interesting survey. That was done by Red Hat last year. They surveyed some of their their users and 94% of them reported that there was at least one security incident within their kubernetes environments over the past year.

That's a huge number of of people self-reporting security incidents. So red hat didn't just stop there. They wanted to find out well what kind of security and incidents what led to be security incidents and so we have this data.

That shows that overwhelming number reported. That was a misconfiguration issue that led to that security incident. And when we dive into well, what do we really mean by misconfiguration?

It's certificates. It's machine identities. So this this is where you know, if you haven't started paying attention to what I'm seeing yet, you know really focus in a one saying that if you have kubernetes teams off doing their own thing.

You know just what was reported by Red Hat 94% of the time there were 94% of incidents teams reporting in a security incident and the majority of those came from configuration issue with their machine identities. So really really important for us to keep in mind. So what what do these incidents cost there have been some others been some other research where if the security incident resulted in a breach of some sort so reach a data might, you know could cost upwards of four million dollars if it caused a system outage and that system outage lasted for significant amount of time potentially up to 15 million dollars.

So so these are incidents that we shouldn't just ignore and you know, if you're part of the the Enterprise infosec team, you know, you should be aware of this and if you're on more on the debt Tech Ops teams, you know, utilize what what your Enterprise security teams are able to assist with because no one wants that to have these incidents occur with this level of cost. So some of you may not be familiar with what's how security Works within kubernetes. So I want to take just a few minutes to provide a little bit of a tutorial so it's very high level.

So if we think about machine identities for cloud native applications and in first, just think about how how does that look? In a classic application where we have a piece of hardware and we want to connect it to the internet and then we want everyone to that connects to that or interfaces with that to be able to trust you know, and authenticate that that piece of Hardware is is what it says it is we use a tee off certificate. Okay, so that's pretty simple.

Well Cloud native application you think of that as well that there is going to be a front end to whatever's however, the application's been constructed and there's gonna be a TLS certificate that the rest of the world uses authenticate. But then if we look at the architecture of what a cloud native application looks like there are a bunch of Standalone services. So this is a major shift from you know, older monolithic applications that used to say right on a data server where with Cloud native, you know, one of the benefits of cloud native of being able to you know, quickly respond to to new feature requests and quickly push out your releases is that it's broken into microservices.

So, you know, if we think about a retail application, there's gonna be a front end that the user interacts with but behind it will be other services that front end application uses such as how do you check out after you've purchase products? You know, how do you manage your car? How do you convert currency?

You know? What's the list of all the products that you want to sell? So these are all separate microservices.

That reside within within the cloud and they have to communicate with each other. So this is really what you know as we think about the complexity of security and Cloud native applications. It shares this with classic, you know, that there's going to be a TLS that's going to connect the internet to you know, your main public view they seem Like microservice, but then you have to think about well, what do we do to protect all these microservices and authenticate them with each other?

And so that's where things like service meshes can be used to help manage that that encryption between those microservices and generally mtls issues for this. You don't always have to use a service mesh, but in general we definitely want to encourage that you have the cloud native application that you're protecting the the microservice to microservice communication with some level of encryption. And we do see many times in the news where companies have not necessarily done this and there have been some breaches as a result of these microservices not being protected with with mtls.

So that has a level of complexity. So, you know, we've got a cloud native application set up with separate applications microservices. They need to have their own machine identities.

So, you know, obviously it's more than just what you would find in your classic infrastructure. But if you take a typical Cloud native application, there might be multiple instances of that might be one that's used for staging or one use for testing or one used in one region in the world and another use for another region the world so you can easily see how these machine identities become. Uh, they they're because there becomes a lot of them.

They they grow in a number of machine identities that have to be managed. So what might have been manageable, you know in a manual method of let's you know, pull a TLS certificate for this application now we're dealing with hundreds or thousands and we have some customers that deal with tens of thousands of these machine identities in the form of mtls. So just the nature of how cloudinated applications are you're going to see that at complexity.

And then we're dealing with things like multi-cloud hybrid Cloud private cloud. And you know, if you look at AWS or gcp or Azure, they're gonna have their own set of tools for dealing with machine identities. And that's that's great that you they recognize it's important and they provide some services for that.

But if you take a larger Enterprise, they're probably not wanting to get locked into just one of these Cloud platforms. So they're dealing with multiple of those Cloud platforms. So now, you know, you've got a different way to deal with Machine identities with AWS a different way with with Google Cloud platform or Azure and it adds to the complexity and so from an Enterprise security team, you know, you need to be aware that there is this huge complexity and that this is going to lead to some problems mainly in invisibility, you know, do you have visibility into all of this machine identities?

If you don't you're gonna necessarily be able to to understand or see the risk associated with maybe something not being configured correctly or something that's expired. You also don't have that same kind of control to where you can ensure that, you know, the same security policies or apply uniformly across all your Cloud made of that applications no matter which cluster their own not no matter which Cloud platform they've been deployed to so it really it does lead to things that I would be concerned with that if you don't have that That layer of control across all of your Cloud native infrastructure. So one thing that a lot of cloud native application teams have turned to to help them manage.

These machine identities is an open source product called certain manager. It's a it's basically it's the industry standard. I've got some stats on showing why it's industry standard but basically allows developers to to request machine identities kind of in a pseudo automated way, but very limited in scope to just that particular cluster and we'll talk about that in a minute but used around the world.

To help them manage those machine identities. so that world that I wanted to talk about and I also talked about some of the challenges that know we're seeing with machine identities and security around Cloud native applications, and let's talk about tips. So first tip is if your teams are not using certain manager, that's that very first thing they need to start using certain manager just to show you how pervasive it is around the world.

5 billion downloads in 2021. It has over eight and a half thousand GitHub stars and there are 300 contributors to that open source project. So just this data Alone says, yes, it is.

Hugely popular. You widely used. I'm not sure why there's still some people out there that aren't using it but they should use it.

So that's tip number one use it. The other important thing to keep in mind is that even though this is an open source project. It was created by an engineering team that's now part of identified.

So we're the ones that have the cert manager developers the cert manager technical expertise. So, you know, I know with a lot of Open Source projects you rely on just community support but identifies able to provide actually commercial support for certain manager and something that we did donate to cncf back in 2020. So again tip number one use certain manager.

Tip number two enforce Enterprise security best practices across all of your kubernetes infrastructure. Most likely you're doing this today with your classic infrastructure. So, you know, you have a way to push out machine identities across all of your it systems, but you need to start doing this with with modern infrastructure as well.

And we see too often that there's a huge disconnect between this kubernetes platform and application teams and what you do in Enterprise security. And and by making sure that you enforce these best practices even across kubernetes infrastructure. You're going to be able to do things like detect misconfigurations, which was that number one problem that that was reported that's going to allow you to prevent outages and breaches.

Here to be able to reduce those security into incidents within runtime because a lot of times we see, you know, there are configuration issues with either the machine identity itself or even configuration issues with cert manager. You're going to be able to identify and remediate major vulnerabilities where they get exposed and this will also allow your platform teams to help with compliance requirements and ensure that they can pass any audits again ensure those enforces best practices across both classic infrastructure as well as modern structure. And one of the things that you can do that you can ask your teams to determine how well are you doing this right now?

It's some simple questions. So, you know, we talked about an Ingress. That's basically the connection from the cloud native application to the rest of the world.

Are all of those protected the TLs, I'm sure they are but are they observable by Enterprise security team? Do you know what their current configuration state is? Do you know if they expired or are they going to expire within the next two weeks?

That should be across every Kubernetes deployment that you have no matter where if it's deployed on AWS or gcp or whatever you should be able to have that single pane of disability because this affects your entire company not just you know, that one one particular group if that TL certificate expires or is misconfigured. Then you know we talked about, you know, the the huge number of mtls certificates that are used within clusters. Do you have visibility into those?

So just ask the question how many mtls certificates are we using within each cluster and are they being managed by certain management if they're not there's risk there and and especially around scalability and deployment. and then as you create and spin up new clusters, are you having visibility to the intermediate Cas that are being used to assign us machine identities to that cluster and then Are your security policies enforceable for that intermediate CA so ask these three questions and if you don't know the answer to it or the answer is no then you know, this is where you really should be thinking about. How do you apply your price security policy?

So that's tip number two. And also he just to the next tip and it's machine. I didn't management a control plane.

So we need basically visibility into all machine identities that are used across Enterprise and that's what I want to talk about with tip number three now. Is that what do we mean by unified machine identity management control plate which work across not only classic infrastructure, but obviously modern infrastructure. It should work across all the Clusters that you have within your your child native environments.

It should work across all the different Cloud platforms that you're using and it should offer these kinds of of features and capabilities. And when you have this you start to minimize those risks that we've talked about before and and this is what we mean what then if I mean by Machine identity management control plane When you put this into place you're gonna end up with basically three take three high level business capabilities one is you have observability to the machine identities across your entire Enterprise. So it doesn't matter if there you know used in classic or Cloud native or which Cloud platform you're gonna have that visibility be able to identify risk of around misconfiguration expiration that sort of thing you're going to be able to control and have a consistent security policy across all those different environments.

So I know that was a lot of material recovery and I just want to get now to kind of the summary in the key takeaways that I would like for you guys to leave with today. So certain related incidents are on the rise, especially for cloud native and you know, if you're on the Enterprise security team, you may not necessarily know what's happening within those kubernetes platform teams and how they're handling security and what incidents have occurred. So just be aware that that these kinds of things are on the rise and companies are self-reporting that they're on the rise.

You want to partner with your your modern architecture teams? And if you happen to be on a kubernetes team partner with your Enterprise security team, learn the language that each other speaks and and you know leverage their skills and their expertise. It's going to make for a much more secure organization.

Certain manager definitely solves some of the problems that companies experience, but it doesn't solve all the problems and that's where a machine identity control plane really becomes important because certain manager, you know, definitely works within issuance and managing of machine identities within a cluster. But you know, we're dealing with most companies are dealing with lots of clusters. And then finally as I just said leverage that machine identity management control plan.

For your time and attention today. Of I really enjoy talking about this topic and you have your phone available. Just scan this QR code.

If you want to download this free white paper. It has some really useful information around how to achieve zero trust using kubernetes certain manager and and istio. Thanks a lot, and I hope you have a great rest of this conference.