Securing K8s Workloads – Sitaram Iyer, Venafi

This is techstrong tv. Well, the great pleasure being joined by Syam Iers. Syam is Senior Director of Cloud Native Solutions at Venafi.

Welcome. Thank you, Mitch. Thanks for having me.

Awesome. Thanks for joining us from Toronto, from the conference, from an adjunct room or hallway, wherever you are. I'm glad you could find some space to be with us.

That is correct. It's better than not sitting on the floor, but that was the, that was the other option. Done that too.

Right? You may do it where you are. Well, please introduce yourself.

Love to learn, uh, some about you, and of course, uh, tell us, identify for folks who may not know about, uh, Venafi. Absolutely. Thanks.

Thanks for the opportunity, Mitch. Um, I'm Si Meyer like, um, I've been with Venafi for about four years now. A little bit of the context to Venafi, the way we sort of, you know, describe Venafi, uh, essentially, um, sort of, you know, manifests itself in think what thing people do on our daily lives, right?

Say you have humans, you have machines, um, and every time human has to access something, so they go and authenticate themselves. Uh, we saw an op huge opportunity purely in the context of machines as well, because that's the one which area which is growing and growing and growing as you can imagine. So machine needs to authenticate themselves as well.

And the defacto standard through which machine needs to acc, uh, authenticate themselves is through something that is cryptographically verifiable. So, um, verify, sort of, you know, invented this category as we call it, machine identity management, uh, which is much broader and bigger than certificate management, so to speak. Mm-hmm.

There is the lifecycle aspects of it, there's management aspects of it. There's the automation aspects of it. Uh, the end of the day, uh, our focus for our, our 600 plus enterprise customers is to be able to help them drive all aspects of machine identity management to obviously avoid any kind of outages or breaches or whatever that might be.

Right. So, so that way they are absolutely confident of the fact that, you know, they do have, um, a, a solid platform in place, uh, to help ensure that, you know, all identities that are specific to machines are managed. Uh, that's the, that's the play.

Um, the, the, the market that bees in, um, predominantly as, as, as you may, uh, as you know very well, Mitch, that, you know, the, the world has changed. I mean, there's a lot of focus on digital transformation, as much as cliche. It sounds a little bit now, um, zero trust, obviously.

Um, containerization, which is something that almost every single organization have sort of embarked on. Um, what that has done is, uh, given sort of, you know, uh, a place for these machines to grow exponentially, and sort of back in the day, we probably thought a machine is a physical server, right? So, but that's not how we see it.

Uh, machine could be a physical server, it could be virtual machine, it could be a container, it could be some software. It could be some function. It could be, you know, something that's, um, running in a serverless environment.

It could be running in Kubernetes environments. Um, so, so the, the exponential growth of all of these kind of functions or things all get categorized under a machine for us, you know, so when we say machine identity management, we are not talking about physical machine. We are essentially talking about anything that needs an identity.

And we know, um, that anything that, uh, that requires an identity, uh, can be secured, uh, can be cryptographically signed. Um, and, and, um, and then I'll jump into a little bit about the, the technical aspects of Kubernetes and how that works. Uh, but that's essentially the, uh, the business that we are in, uh, to help, uh, manage machine identities.

And you said several things there we'd love to unpack. We can't have time for all of it though, but it's interesting, you know, a lot of times when people say machine identity, or at least what they interpret that as, they kind of think of as a web server virtualized or whatever, as a web server certificate is sort of like the very simplistic starting point, if you will, or maybe security around APIs and things like that. But in a cloud native Kubernetes world, right?

We have workloads that are traversing, you know, many, many different things. Machines, if you will. You know, everything from an iot, OT device, a serverless application, or multiple components at the edge, microservices that are living anywhere and everywhere in, in whatever number of instances.

And it isn't, you know, kind of one memory space we're protecting or even one set of, um, interactions that are happening across microservices, right? Oftentimes mm-hmm. We need to secure individual workloads that are happening anyone at any one time, and like automation comes in because we're not gonna have a person issue a digital certificate ev every 10 seconds or 10 minutes, or 10 hours whatever, for all the different things that are happening.

So it's so dynamic and happening so fast and changing, you know, you have to, uh, think of security differently in a cloud native Kubernetes world, Absolut, correct? Yep. Am am I on Absolutely track there.

Yeah. Oh, you're absolutely right. And actually expand on that ex same, uh, example that you just took.

You know, uh, end of the day, every organization, uh, is serving their customers, right? com. That's an experience for my consumers.

Let's say for at, at your organization, you say, you know, anytime somebody comes in, this is their landings place. dot com landed somewhere. It's not the end of right.

So it provided you some experience that you are beginning to experience now. Uh, because behind that scenes, you may have hundreds of microservices that are deployed run, because, you know, somewhere sometime you decided that, you know, we need to be an organization that has cloud native architecture, cloud native thought processes. We are gonna containerize our applications.

Everything through that we build is going to be microservices based, um, architecture. So you build this application that says, all right, so I've got a product service, I've got a catalog service, and I've got a card service. I've got a payment service, I've got a recommendation services, I've got ad services.

You know, all of these are potentially, you know, in an organization built by different teams. They all need to be wired together at some point, which essentially translate to that experience that you land with. Uh, but when you look at these as individual microservices, they're functional, they're testable, they are scalable, they're all, everything that you think of how it should be in a cloud native world.

When you put all of that together, you're providing an experience. Now you could say that I secured this endpoint that's coming into the service. Everything else I wanna implicitly trust or in a more security conscious organization say, well, that's not the only one thing that we wanna secure.

We want to secure every single line of communication between each of these services because these workload card payment service needs to be secured. This workload card, currency service need to be secured. This workload called card service need to be secured.

Mm-hmm. And when those services talk to each other, they should talk in a mutually authenticated way. Cause we don't, we don't want payment service to be called by somebody else.

We want that payment service to be called only by card service. I'm making up an example. You can imagine That's, that is the correct or appropriate or authentic payment service, Authentic payment service.

We're connecting to one, the one you're connecting to. Right. You know, you don't want that.

Yeah. So, so in that case, and because you've gone this cloud native world or the cloud native architecture that you adopted, you have designed it in a way that, you know, this payment service can be upgraded, updated on a daily basis. You've got pipelines in place that are automatically upgrading it.

You've got, you know, development teams that are just focused on writing third without worrying about, you know, how these things are deployed run. Because there is a platform engineering team that is managing and running in this context. We forgot one persona, which is the security persona who are essentially accountable from an organization's perspective if something were to go wrong, like, you know, they are eventually the people who are accountable.

So from that perspective, they look at it as how are you securing the communication between currency service and the payment service or a card service and the payment service? Are they mutually authenticated to each other? Can't say that, oh, we secure that.

com. There is a digital identity. That thing was valid for 12 months or three years, or 10 years, or whatever that might be.

Everything else is good, doesn't work these days, because, you know, we want that everything to be, to be identified. Um, you know, simple example that I can take is, you know, um, um, and I'll get into a little bit of a specifics here. Um, these workload identities that we talk about are something that can be derived automatically, right?

So you don't have to work hard to derive an identity for this workloads. Um, and that's where something like, you know, spiffy comes into play. Like, you know, I could say Mitch and me are talking on Zoom.

Can we attach a spiffy ID for it? Spiffy colon slash slash zoom slash mitch slash sit around slash meeting. That's a spiffy id.

We iden we attached an identity for our own meeting. The same way an identity can be attached for the workload. Because when a workload is deployed in Kubernetes, we know where that workload is.

So, which means we can derive an identity for it. The next step, which be to take that identity and cryptographically sign it, which makes it a cryptographically verifiable identity for that workload, then use that workload identity to talk to other workloads have been done and gone through the same process of being attested with an identity. Where comes in play here is that, you know, those identities end of the day needs to be signed by somebody.

And that signer is typically an organization within, uh, within the larger organization, a security team who manage, uh, all of the ca infrastructure, all of the PK infrastructure, all of the machine identity infrastructure, as we said, as we just talked about, they're responsible for those cas because those cas are, are, are the crown jewels of the organization, right? If you mis they are, they're typically, you know, sometimes stored in a vault, they're stored in sms, you know, they're stored everywhere. Uh, so somewhere you have this one abstracted level and intermediary or assigning certificate or intermediates that are, is, uh, that are managed by the security team, but then zoomed by the platform engineering team to sign it and validate.

So that's how it all place together from a perspective of how we need to secure it. But end of the day, uh, when we talk zero trust and architectural patterns around security, it's not just the K north south that we wanna secure. We wanna secure all of the EastWest traffic as well.

And that EastWest traffic, uh, also ensures that, you know, how communications happen between services. Uh, And you kinda have that fourth dimension to it too, which is time, right? Because time, a lot of these are what they call short-lived, uh, crypto trips, graphic connections, right?

I get Absolutely those four loads. Our Zoom call might last 20 minutes, but that could, it could be a 22nd correct. You know?

Yeah. Yeah. I think, yeah, I think we, we, we also put a lot of emphasize on the exact thing that, and that's a conversation that I have very regularly.

Workloads are highly ephemeral. Mm-hmm. So, which means the identity that is associated with that workload are also highly ephemeral.

In some ways that identity could be an hour long, or an identity could be, you know, 30 minutes or, you know, identity could be seven days. It is not the, the old world of I have a digital certificate that is valid for 10 years, doesn't work anymore. Cuz you know, it's just, you need an identity for a workload.

As long as that workload is running, if that workload doesn't exist, there is that identity should be gone. It shouldn't, it shouldn't exist, um, at all. So that's, that ties into the ephemeral nature of the workloads.

I think, I think of it as, um, that's the difference between, um, identity management o of environments and, and the l actors within that people machines, et cetera. Um, but workloads are different because, I mean, yes, environments can be ephemeral to a degree, you know, a terraform and things like that. Other things that can dynamically change your environment, but the workloads themselves can change significantly or happen concurrently.

Or you, you, you need to tie the, uh, cryptographic, um, identity of a specific workload. Cuz that's what you need to be able to attest to. Absolutely.

The validity of that. Yeah. Actually, even before that, there is a sub step, right?

Say somebody is sitting there and writing code, right? Mm-hmm. They're writing code every day and they haven't stopped because, oh, that payment service is deployed.

We have done with it. There is always an improvement. There's always some things that they're developing on.

So, and from a, from a typical lifecycle perspective, you can almost imagine a developer writing code commits a, to get, there is already some sort of, um, security aspect tied into it because most organizations say you cannot commit code unless you have, uh, a developer certificate of origin that says you are who you are. That is committing code. That code gets compiled, it gets posted into a image registry, and that image registry has an, as the image of this payment service, right?

Click, click, for example, a docker image. That image also needs to be signed to ensure that, you know, when we all have this experience, I dunno if you're using a Mac, sometimes you download something and then you double click it says, I cannot open. And then you right click and say open.

This is developed by unknown developer. Do you trust it? Click open, and then you say open, it becomes trusted.

So ideally in a, in an enterprise, we want to ensure that, you know, there is, that code is signed by authority that is managed by the security team. So once you sign that, that container image that is in the registry is used for deploying that workload that we talked about, like maybe service becomes a reality in a container in this one. So at some point, if we sort of, you know, look at, um, look at the, the provenance of this one, we should be able to tie back this contained this workload that is running in cluster A in a w s or Google Cloud has his provenance.

That can be tied back to the image that was signed by X strong, which is on registry X with a source code that is available here, signed by these developers, which contribute all back. So then you have this, you know, almost like a, like a, the, the whole aspect of, you know, all everything that people are doing with supply chain, right? So that's essentially fits into that model.

So you can always go back and tie back its provenance back to the code. Um, and that run time, there is an identity associated with it. Yeah.

I'm curious, uh, um, I'm always curious about the intersection of security, engineering, security world, and of course software and, and the, you know, DevSecOps, yes, there's shift left and all of that. But I think more importantly is how are, how does security engineers view this aspect of identity management? Because machine identity probably met more machine, you know, machines literally and virtualized.

Um, but in the Kubernetes mo uh, uh, Microsoft's world, you know, we're talk, having a much different conversation. Spiffy exa Yep. As a framework for that.

Yep. Um, what kind of security engineers typically are talking to software and software infrastructure, uh, developers and engineers about how we're securing the environment and the workloads? Are people that come outta software, or is it at security engineers that have kind of delved into the Kubernetes software architectural world?

What do you see happening with that? It's a, it's a very interesting question. I think, you know, uh, I, I'll tell you cause my experience has been, um, very, very, very, um, so it is a continuous education point because, you know, you can, in some ways you can look at the, the classic security folks that have been in the organization for the last X number of years, right?

So they've been around for a long time. So there is always a security organization. Platform engineers are somebody who started off what, 5, 6, 7 years ago and evolved from VMs to, you know, things like Pivotal to all the, all the various different containerization from Docker to Kubernetes.

So that's, that, that world has sort of, you know, slowly evolved in the past 6, 7, 8 years. Security has always been there. So we go to talk to security and talking to them in the context of workloads and Kubernetes is a hard conversation because it's, it's doesn't sort of, it, it's not in the normal sort of a conversation that they have, right?

So when you talk, oh, ca and p k i and this, that's understood. But the moment you say you need to secure a workload, what's a workload? And then you need to, yeah.

So when sometimes the conversation is more educational, but on the other side mm-hmm. Platform engineering. Um, I mean, five years ago, if you asked me to secure that workload, I had a easy way.

I would run open s ssl, generate a self sign certificate, attach it to the workflow and say, I'm done. Cause I didn't know anything about, uh, compliance and ca policies and organization manage cas and everything. So, so that, that, that security engineering or the security teams and the platform engineering and the developers.

So there is multiple levels of sort of, you know, different conversation to be had about the value that they get for automating and thinking about security. And this has been a constant education thing for us. And we go to security team and say, your security team, you've been managing this aspect of machine identities for 15 years.

Do you have visibility to all of the machine identities in your cloud native environments? Very often the answer is no. And then we talk about, you know, what should they be doing to get that visibility so that it can manage.

On the other hand, platform engineering say, well, I didn't, I, I don't know about security. All I needed was an identity and going through the route of getting an identity from a security team would take me two weeks. I'm not waiting two weeks.

And that's your point about going fast, right? Our workloads get deployed every day, and I need an identity for it. So I just figured out something on my own and I got an identity for it.

On the other hand, developers which are developing the code, they say, I mean, it just needs to fit into my pipeline. As long as I can automate something, I can consume it. So, so, so most of our education has been around one, to ensure that the security engineering have a good context to ensuring that your workloads span or your, the nature of your workloads have changed.

It's not that Apache instance that ran on that VM that you need to secure 10 years ago. You may have hundred containers that do the same thing running in Kubernetes clusters. So from a work perspective, you're still accountable for securing those workloads.

You do have a set of services, set of machine identity services that you can roll out to your platform engineering team so that they can utilize the platform that you are managing from a ca perspective and still get the best benefit out of it. And that's where cert manager fits in. Cert Manager, essentially, I see it as that bridge between the platform engineering, which they understand very well.

Cert manager, oh, it's a certificate controller runs in Kubernetes. Great works for me because I live in the world of communities. Mm-hmm.

Security teams says, well, we have some consumer called cert manager, which needs identities from the CA platform, or the ca issuer or my internal PKI infrastructure that ma that, that, that issues certificates. Well, let's connect them both. And then you have security team who have the ability to govern and manage certificates or machine identities.

And from a platform engineering perspective, all they did was connect to the platform. Um, this is an ideal world, but to get here, to your point, it takes a lot of conversations because mm-hmm. The value each one of them sees very different from the perspective of, you know, how they, uh, how they get it.

So, so we are, we are in a continuous education mode to, to to, to a short answer to that. And, um, and I think, you know, in some organizations are doing really well on that, you know, in terms of, you know, trying to get, try where, where there is, uh, a good relation or a good working relation between security engineering and platform engineering. This is an easy conversation mm-hmm.

Where security folks have controls that have not sort of, you know, adopted to modern cloud native development. Right. It is typically a hard conversation.

And, and I'm, I'm, I see both and I'm trying to bring, uh, a middle ground is, is, is is always a, always a, a conversation. You know, I think some of the, um, some of that middle ground is, is all the things that security teams know about in PKI I and certificate management and how you use them, why you use them, you know, using external certificates authority. Yep.

Uh, ca for that. Um, the way I explain it to folks, or at least start the conversation is essentially what happens in the Kubernetes cloud native world, the network is inside of all of that instead of the outside that connects to it. Right?

That's what all these microservices are talking to each other. They're over, over API's, right? Absolutely.

They're all being managed through services that are accessible through, uh, digital communications APIs. And that's what we have to secure. And we're gonna do the same thing we, we do in network elements and communications happening across networks to all these things that are happening and changing a lot inside this Kubernetes world.

So while that may look a little mysterious and complex, you know, a lot about already about digital certificates and, and certificate management and why, and what and how, and what we're really doing is applying it to this world, but also with a lot of automation. Cause it needs absolutely Needs change. Yeah.

More quickly. And that's then you've got a good starting point for the security engineer to have that conversation with the platform or security or architect about this. Okay.

Help me understand. Okay, great. All right.

Let's get into the z your ca. Okay. That's where you're going to, that's how we rotate keys.

There's how things, you know, exactly authenticate, here's how they get torn down. All of those things. And at a station for all of that security people go, I know that.

Now I know it. In a new environment. Yeah, yeah.

At attestation is, is is specific. I think that's where we start. I mean, I, I, I'll sort of, you know, expand the conversation.

So we were talking about Spiffy a little bit ago, um, a little while ago talking about, you know, how that plays into a role, right? So, but, but there is another, there are other implementers of Spiffy, which is Spire. Um, and at testers play a huge role going back to the same example of, of that card payment currency service.

And if you sort of, you know, think about it, if you sort of expand that to say, well, by the way, card service runs in, in my Kubernetes cluster that we are managing in our data center. Payment service actually runs in aws currency service runs in Google Cloud. And now you sort of, you know, think about, you know, how they have to talk to each other from an identity perspective.

We ensure that an identity is issued for each one of these services. Mm-hmm. But that doesn't preclude how you can connect to them because mm-hmm.

Google says, in order to talk to me, you need to have a workload identity, which is a concept. Yeah. Google has AW says, oh, you have to have a proper IAM role and the permissions associated to talk to that workload.

And that's where these Spira testers come into play, which is very powerful concept, essentially to say that, well, this identity that is in Kubernetes in cluster A can be attested using or translated with the workload identity that is for Google Cloud or an IM thing that is for, uh, you know, in, in aws. So there is an AWS tester, there's a Google Cloud, arter, there is a physical, something else at testers. And all of these arters play a huge role in the world of Spire, essentially to ensure that each service can authenticate itself to each other while you focus on the primary requirement of providing value to customers.

Right? Don't worry about, you know, how to, how to connect to aws. Well, you have, there are things that you have to do AWS specific to connect to aws.

You can't do something generic. It has to be AWS specific. You have to do something in Google Cloud.

You have to do something specific to Google Cloud. But from a generic identity and, um, authentication perspectives, you want to design something that leverages all of the capabilities. Um, and then even in that sp as you attest, there is the at tester, and then there is also the, the workload APIs.

Cause each of these workloads themselves need to figure out how to leverage those identities, right? Like if you say, if you call me aws, you have to identify yourself in some way. If you call Google Cloud, you have to identify yourself in some way.

So all of that has to be sort of, you know, implemented. It's a very interesting space because, you know, from our perspective, we also see that, well, if you are going to identify an issue using sp, so we built an integration into sp uh, SP has, the SPY is extremely extensible, pretty awesome tool, obviously. Um, there is an upstream certificate authority plugin, and you can see where I'm going with this.

Mm-hmm. The moment you say a ca plug-in. So we can integrate into that, which means you can integrate back into our, back into the security organization and say, and that's the kind of conversations we want to have with security team as well.

You will hear new terminologies, you'll hear Spiffy, you'll hear Aspire, but they all need an identity that is managed and controlled and governed by you. So there is integrations available, cuz many times security teams are more focused on, I'll give that one year certificate for that web server and I am good. But we say it's not good enough.

You need to be able to enable these teams to be able to sort of, you know, be self-service, have that automation framework in place, you have to have this, um, you know, some sort of assigning certificate mechanism through which you can, uh, allow or facilitate those, uh, workloads to be signed on their own. So, um, and I'm pretty sure you've encountered this in the past, security teams will never issue an intermediate to somebody else, Hey, here's an intermediate, go do whatever. That never happens from where we saw an opportunity with the security team is say, that is a basic requirement for workloads that run in cloud native environments.

They're not gonna gonna come all the way to a ca for every single workload to get a certificate. That duration, that latency is just not acceptable. You need to have intermediates where the workloads are.

But by the way, security team, we will provide you capabilities and, uh, functions to be able to manage that. And that's essentially where, um, just as recent, I don't know if you had a chance to look at it. Uh, we announced during UHON in Amsterdam, uh, a new sort of a functionality or, uh, a product addon to our, uh, our capabilities that we call Firefly.

The idea is that this intermediate is now a managed intermediate from a security perspective. Um, otherwise typically from a platform engineering perspective, I would just go, can you gimme an intermediate? And the first answer is no second answer, no third answer, no.

So I mean, every time it'll be a Managed one. Hmm. Okay.

Managed one. It's okay, we can talk. Yes.

Actually that's, I think that's, that's Easier. Easier in hierarchies. Yeah, exactly.

CS are in hierarchies for a reason. Yeah. Yeah.

Well, you can create as many hierarchies as you, you can gimme one level above and I'm perfectly fine. com. Like what stops me from, I mean, it, it won't be usable outside on a browser or anything.

Right? But internally, I can issue certificates. And I think that's where we talk about policies and you know, how you can govern them and, you know, ensuring that, you know, you can only do things at a certain certain way.

And, um, all of those policy controls that we talk about, I guess that's the difference, uh, between identity management and p policy enforce policy Policy enforcement. Yep. Yep.

Well, unfortunately we're gonna have to I understand to Keep going. Yeah. Um, I'll talk to you about an event we've got, hopefully we can, uh, get you back on and, uh, be part of Cloudnative event that's, uh, coming up.

This is Hawaii. Yep. Absolutely.

Been fascinating to talk to you Citra. Yep. Uh, thanks for joining us, especially from your, your dynamic travels and, and undisclosed locations, wherever that may be.

Yeah. Wish you safe travels and we'll see you back again. This was, uh, Citra ier who is senior director Cloud native solutions, and where can folks go find out about Firefly and check out some of the other great capabilities that Vany offers.

Yep. com. That's the, that's the quickest way to come and, um, you know, understand, uh, everything about, you know, what we do at Bey, more information about Firefly, how we can help support, you know, all of your communities environments.

com. Uh, It's a great website if you're a security engineer, kind of getting your head around the Kubernetes microservices cloud native world. It's a great site to look at because you're talking all about how you secure identity management working in that environment.

So, good resource. Thank you very much. C good talking with you.

Thank you. We look forward to having Absolutely. Thanks Mitch.

Thanks for having me. Yep.