Five Disagreements You Should Be Having About Running Kubernetes Everywhere | DevOps Experience 2022

Hello folks and welcome. This is five disagreements. You should be having about kubernetes and how to solve them.

If you're here for something else. You're gonna get this talk. Anyways, let's go ahead and dive in and we're gonna introduce ourselves tell you a little bit about who we are the company we work at so that we have the established the credibility for why we're giving this talk, but let's go ahead and get started Bill.

Why don't you start? Hi. I'm Bill letting him on the CEO of Fairwinds.

I've been a CTO for 20 years before that. So I have a good perspective on technology, hopefully. All right, and my name is Kendall Miller.

I'm a technology evangelist here at Fairwinds. I've been around for a very long time seven and a half years of this company seven something that's been a long time. Anyways in the kubernetes world for a long long time arguing with all the people about all the things Elisa.

I'm Alicia Hebert VP of engineering operations at Fairwinds. So I received the software and services teams. I am pretty disposed to argue about everything.

So this is right in my wheelhouse I've been doing Uh Cloud Consulting and software for 15 years now so said for arguing with my three of my faves today. Glad to have you here and Andy go for it. Yeah, I'm Andy Superman.

I'm the CTO of the new in kubernetes for about seven years. So I'm predisposed to argue about everything related to kubernetes. So this is also very much my wheelhouse and I love arguing with Kendall about everything.

So that's Fairwinds for about four and a half years. So I've been doing that for a long time, too. I think you argue with all of us.

I think you you just have a rule that you must argue with Kemble whether you think I'm right or not. You'll still learn. Yes.

Um, so real quick Fairwinds is a kubernetes company. We make it our mission to provide security policy governance help customers ship software faster with kubernetes. Everybody's using kubernetes.

Everybody's afraid they're using it wrong. We give people confidence that they can use it correctly across security reliability efficiency big cost savings bits right now, which is pretty relevant in the current economy. And that's why we're here today to talk about this topic and I'm gonna stop sharing slides so that we can have a discussion and dive in and let's start with the five disagreements.

We've got five disagreements. We're gonna talk about and we're gonna discuss them as a group and hopefully keep this relatively lightweight and just go smoothly through this so number one you Should be giving your CEO access to your kubernetes environment and Bill. Do you want to start with the pro?

Why should we be giving you all the access or not? Why I certainly appreciate the vote of confidence, but quite frankly. I haven't written a line of code and over 20 years and I think frankly that will be a very dangerous thing to give me any access to any kubernetes cluster.

So while I appreciate the thought I think probably the larger question should be should we be giving developers access to kubernetes clusters, especially production clusters and/or kind of you know, how do we need to balance accessibility versus having the right guard rails in place to make sure developers are not going to muck things up once they have access to the cluster. So, I don't know any thoughts on that one, Andy. I have very strong feelings about that.

I think Kendall was was gonna ask another question here. Well, I'm curious like Bill though. The the problem I have with that is sometimes you sell stuff.

You're the CEO, you know, lots of you know, lots of people some of our customers are our personal friends of yours that found out about our software by way of that. Let's say something's going wrong in the middle of the night. They call you you told me everything was great.

It was gonna be amazing and I'm having this problem get in there and shut it down shouldn't you shouldn't you be able to get in there and just turn things off? Yeah. I know that that works for like 10 person startup.

But once you get any scale, you really don't want to be doing that and okay. Well, it's great to let the CEO demo different features. It's not a good idea to give them access to a cluster where they could potentially break things and really screw things up.

Okay. Okay. So then then back to Bill's question Andy, I mean, do you even give the developers access?

Yeah a great question. I mean I first want to say I have written a line of code in the last 20 hours and I argue whether I should have access to our community's closer. So but as far as getting developers access, I definitely think developers should be enabled to be responsible for their applications all the way through production Enos safe way.

So giving them either the ability to deploy their own code via githops or through, you know, controlled deployment processes with governance in place that don't allow deploying the wrong things or giving them, you know read only access to diagnose problems in their kubernetes clothes kubernetes cluster. I'm a huge believer in enabling developers to you know, own their code from beginning to end because they're the ones who wrote it. They should know how it's deployed.

so SSH access for everyone No SSH accessor. No one that's for me kubernetes. We don't need exercise anymore.

So come on. Well, yeah, I mean so at least along those lines like You're an executive at this company aren't all the executives technical that we should just they're probably technical enough. We'll give them access.

I mean if if this if the CEOs credit card is the same credit card, he uses to buy T-shirts off of Amazon. You don't think he should have access I mean, I think everybody should have access just throw caution to the wind and you know just crush your fingers and and hope for the best. I mean also the question was should your should your CEO of access to your clusters, but your first line of questioning was should your CEO have access to all of your customers' clusters and you know, I think to Andy's point it things should be constrained to the sort of necessary folks and in my case coming from the business side.

I do not qualify as a necessary folks. Okay. Okay.

Well, let's go on to number two. Uh, no the second this or agree disagreement argument. You should be having about kubernetes.

You should be deploying all services into your kubernetes environment and I mean all of your services including your databases and including your WordPress website and including your self-install of Windows Exchange Server 2003, that's still running your email. I mean I'm gonna take the pro on that because you have kubernetes you might as well just put it there. Right?

Do you want me to run screaming from this this webinar is it would just say do you change 2003? I was I'm getting scared here just lift and shift everything right Kendall. Yeah.

Yeah. I mean if kubernetes right see Bill gets it bill and I understand page. I mean I did ride a blogger article called shoving them on a list into a container probably about six years ago.

So there's some validity that but I think the more important question that you should be asking is should you be running those things yourself should you be running your database should you be running your git read your you know SVN should you be running your cicd process entirely and my answer is no you shouldn't be if it is not your core competency in the thing that you do as a business that you sell you should not be doing it. You should be paying somebody else to do it. Until you hit a certain scale at which that's not economically feasible anymore.

But wait if we own every last bit and have a strong culture of if it's not written here. We don't use it. Then Elisa gets a lot bigger team and at least it don't you want the biggest possible team riding the most code and maintaining everything possible?

Well, I've heard that the metric that you should use for productivity is lines of codes. So I think yes definitely go on that direction. Yeah, emotionally that there.

Okay. Okay. So so real quick, we'd probably shouldn't be deploying everything.

There's probably some things we don't need to own some of the things we own like our database. We probably do need to own right handy. Do we put that in kubernetes?

No, and you shouldn't you own the database but you should use a managed service like RDS or you know, Google's equivalent of that. I just forgot the name of but yeah Cloud SQL, I think yeah, there's also like third party ones that do something similar. Yeah.

I haven't run my own postgres since accepting a doctor container on my own laptop for testing in years because RDS Does It Better. Yeah. I mean, I don't know there's some amount of like we adopted kubernetes because it was the future.

And that's what somebody told me. So let's just put everything in future is how I would probably go about it. Um, okay.

Okay, let's move on number three developers should be in charge of everything all the way through to production and he's already argued the pro for this. You know, let's let's go to the con Bill you've been in tech for a long time. Let's talk about back when everything was super divided and there was a development team and all they had to do is write code and throw it over a while.

It wasn't a problem after that wasn't everything better. Yeah. I know when I started out we had these things called Mainframe so, you know person would just yeah throw COBOL apps at the Mainframe when you know all the sudden done but no I always works right we never had any problems.

Yeah, that's right. Throw it over to Q&A or QA and then throw it over into the infrastructure. You know it team and don't worry about it.

No, I I do think I would actually argue the flip side where you know developers need responsibility for ownership of you know, end-to-end these days because of the complexity of the technology. They know how the app should run but they need guidance from the infrastructure team around, you know best practices for what they need to be aware of as a deploy containers into kubernetes. So there's a knowledge there and a handoff process that needs to happen.

You want a free up developers to allow them to focus on developing the application and value add but you know, they do need guidance in that. Kubernetes, you know frankly is a complex technology, you know, they need to be aware of how to use it effectively. They need to have the right level of permission to see kind of what they need to see but you know, they don't need to be owning the infrastructure.

So there's a balance you need to strike there between the two Yeah. Is in order for developer to own things all the way through to production. They need good guard rails around what they're deploying.

They probably need to write competent logs and they're probably bad at writing confidence log. So when bad things happen, they're not even gonna know what happened right? Because they're probably bad at it.

They probably are not using tracing because the develop the Ops Team probably didn't put that in place. I mean, you know, what if I have to turn off the admission controller because I have a really really good reason and I just need to go around the guard real just this once to deploy this one thing to fix that one thing for a customer like like I don't know isn't owning things all the way through to production gonna cause problems. No, Andy I mean, I think Bill answered this question already, he said we need the enablement team from of the enablement function of the infrastructure team to help with those problems, right?

It needs to be a process by which they have a feedback loop that they can inject themselves into and say hey you need to fix this to get through the admission control or we can make an exception here because it's necessary but putting those guardrails in place and allowing that feedback loop to happen with Ops in the mix is important. Yeah, okay, and then when there's a problem, you know exactly which Engineers responsible for it for those blameful post-mortems that we plan on doing. Yes, definitely so that we can point there's always one directly and that that team yeah exactly.

As long as we're clear. Oh, okay, you need that clean handoff process, otherwise everything, you know quickly degrades, you know, as soon as you kind of blur the lines there, that's that's when problems start to occur. I mean, I guess I'm being so overly sarcastic that maybe it's not always obvious what our actual opinions are and Bill's bringing us back.

So I appreciate that bill. Another fourth fourth disagreement here. Your CTO should retain access to SSH into individual containers.

I made a reference to this previously. I actually think you can you know, I've said this before to you Andy if we just remove them off of all of the container base images then you know, no real hacker can get anything done anyways, and so I wouldn't be able to function if you did that. Yeah.

So should our CTO retain access to SSH and individual containers. I mean, at least you already touched a little bit on give us a little access as possible except when they need it is is this that situation I mean, I think that is a hundred percent of question for our CTO since at this point that they have. Landy take it.

No, we don't want access to get into the containers. I mean executing into individual containers is a bit kind of pointless. Anyway, they're ephemeral immutable.

They're gonna go away eventually. We should have, you know, tooling around them to be able to diagnose the problems. We should be centralizing logging and things like that.

I don't want to access to the containers that being said I have access to all of them right now. So I don't know maybe I'm not the right person answer this question. Do you think the answer to the question depends on the size of the organization though?

Right? If you're 20 people or two people or 200 or 2000 is that does that answer change based on the how high up? You know problem solving initiative needs to go.

Absolutely. I think the role of the CTO changes drastically from you know, five one person to five people to 20 people to the 45 that we are now and then beyond that so definitely changes in your access to things should decrease the further up that goes increases, but but Something wait now now I'm confused. I've been super sarcastic One Direction and now I'm gonna come back very seriously and say wait a second in kubernetes shouldn't no one ever need access to SSH into a container isn't the entire concept of it.

We'll just kill that container instead. Is there ever a reason why it would ever make sense? Like I mean, I guess maybe in a small environment where you have something small going on maybe in development you want to SSH into a container and change something so that you can understand what's going on.

Is there even a use case where that makes sense or would you just kill the kill the pot or kill the container itself and bring a new one back up. There are times when in trouble shooting it is still necessary to exactly do a container. Ideally this would happen only in your Dev or staging environments and not in your production environment.

But I've also had to SSH to nodes to diagnose why I keep it won't start or something like that. So there are times. When what we call break glass access is necessary.

But in general what we try to do is put guard rails around. That so that you have to you know request that access or it's audited properly in such a way that you know, it doesn't happen very often. Ideally.

You're right shouldn't be necessary. We should just you know, kill the Pod move on or get the information that we need from logs and tracing and metrics. observability as it were without incriminating a specific company Bill.

Have you ever had more access than you wish you had? Is there ever times where you had access to things that you were like, I don't need access to this. Oh, definitely, I think by default companies tend to be overly permissive with access to different environments because you know, you always want to kind of you know, buy us the organization towards speed of you know development speed of getting things done and that implies probably a higher level lack of access and what you normally need and so but you know with that comes certain pitfalls and you know, certainly if people you know inadvertently do the wrong thing, then you're exposing yourself.

Okay, that's interesting. Yeah, I mean I can think of a couple times that I have had access over. Oh overly been given access to things.

So that's entire. I remember a time in the history of this company where you had access that we're over and above. Yeah, really not want me to have Access and like please take this away from me because I don't need it.

That's exactly that's exactly right. Okay our final one. And then I think we have some time I can dive into a brief demo of what Fairwinds actually builds for software and show how it's relevant to all of these things.

But for our last topic you shouldn't worry about your Cloud cost and Elise. I'm gonna start with you on this one because you don't care about us, right. So like we move to the cloud because somebody told me it's cheaper than owning my own infrastructure always so just like Donuts right.

I mean is that the attitude that we should have? I mean a hundred percent you should have that you should not pay attention to any of your costs and you should just hope for up into the right with your revenue and and that way that solves all things. I mean, I am never pingis in the background for anything.

Yeah under percent would not hear from from Mary on that at all. I think it's important to understand the unit cost of things. So whether that is delivery of a service or delivery of a piece of software or you know, per node per cluster per customer per Revenue, right?

I think, you know figuring out what the appropriate unit costs for things is for your business and the way that you are set up and the way that you intend to scale and grow. is important they're certainly, you know, some some guidelines out there, but they're not going to be the same for everybody and and knowing how your Cloud costs fit into your overall cogs for the way you run your business is I think something that all engineering leaders should understand. I think it's a place that we could use a little additional expansion in terms of perspective.

Okay, but I mean isn't it impossible to figure out unit costs in a multi-tenant shared environment where all of your customers are in the same bucket. I mean isn't that just literally impossible to do? I don't know.

I think I asked you to do that yesterday so you that's what I'm referring to it man. Have we got software for you? Is that our YouTube me up here Amy.

I mean partially yes, but also like this is the discussion. We haven't we haven't done early and it's hard but it's important and it's really hard and and it's never gonna be as accurate as you want it to be but you can get a reasonable approximation if you if you try at it, I think. but The effort has to be has to be known like don't chase every unit cost because that's you know, diminishing return, right?

Yeah, but you know these days I think it's even more critical given that we're it. Looks like we're going into you know, certainly a mild recession. If not a full recession.

Yeah, you have to be cognizant of what you're spending and as many companies are trying to move more more applications to the cloud. It's even more critical to know kind of what you're spending per application. So you're doing it responsibly and you can you know kind of forecast where your costs are going as you move, you know more applications over to the cloud.

So I think you know totally makes sense to move more to the cloud that you need to know, you know. How your costs are evolving as a result of that. Yeah.

Oh with that in mind. Let's go ahead. Brief demo.

So we've talked a bunch today about different disagreements about kubernetes different really as it relates to access as it relates to cost as it relates to guardrails what you give Engineers access to Etc Fairwinds insights to software that Fairwinds builds to enable teams to shift policy and responsibility left to developers that they can build software ship software all the way through to production understand what the guard rails are. I as a developer. I'm not a developer, but if I was a developer and I was owning things all the way through to production I want to be stopped.

So I'm not making stupid mistakes a long way. I don't want to wake people up. I don't want to cause problems.

I don't want to create security incidents. I don't want to overspend but I don't know how to avoid those things particularly in kubernetes because it's a wildly new Paradise for the most part. So that's where Fairwinds insights comes in companies use this to make migrations to kubernetes to build policy and guardrails around kubernetes so that they can deploy it with confidence.

Across the organization I'm gonna walk through it real quick at a high level. I'm not going to get into every detail but I'm gonna show you a little bit about what Fairwinds insights does why it's related to this when you log into Fairwinds insights, you're going to get an overview of a couple different things that are going on. And in this view.

I'm looking at every cluster across my organization including my favorite cluster, which is sadly offline, but you can see all these other clusters across the organization we get a health score for each cluster and overall health score for the organization. I can click into any one of these clusters and it gives me an overview of how that cluster is doing. So I get a health score for it including all the different things that we're checking for in Fairwinds insights where action items are introduced into the Clusters above the line where we're knocking them out of the Clusters below the lines we can see things should be improving over time and this is a dental cluster.

So it's not particularly exciting but in theory that Health score would be going up into the right and a positive way. We get some high level overview of what the cost looks like for this cluster and I'll get Into that in a few minutes, but this is some of the overview of what am I doing? Right am I doing things right or not?

And if I'm if I'm a platform engineer, I want to go put in place a bunch of guardrails. So where are we getting this? All from Fairwinds builds a bunch of Open Source tools to help you with best to help you with best practices for kubernetes.

So Polaris has been a tool that we wrote a long time ago checks for common misconfigurations and kubernetes workloads things like is the container running is root. Did you over permission something? We also check for out of date Helm charts deprecated apis.

We have a tool that gets resource requests and limits just right but we pull in third party Best in Class open source as well such as trivia to check for those container vulnerabilities. You don't want to be deploying CVS into your cluster who bench Coupe Hunter then we have support for Opa so you can write custom policy. You can write custom policy a couple different ways, but oppa is kind of become the standard so that all rolls Under this big list of action items where we can see every single thing that's going on and we can filter them by severity critical high medium.

The critical ones tend to be security related. I can click on this and see we've deployed something with a known vulnerability. And so I here's what I need to do is go in and update certain manager and it's going to tell me how to do that.

So importantly I don't have to be a super senior kubernetes engineer to understand how to go knock off an action items. But also if you're like me and you are a platform engineer and you're responsible for a large organization with lots of clusters seeing this, there are 4,19 results and that is wildly overwhelming. I don't want 4,19 things that I need to fix.

So what I want to do is put in place guardrails shift that left so that developers are knocking off those things as they're deploying workloads into the cluster. So we have integration in the CI pipeline. This is what that looks like.

Here's an engineer deploying something that they shouldn't be applying and they get right and get They're gonna already live anyways, a list of all the things that are wrong and they can click on this view the full report see it in Fairwinds insights. It loads it up in the GUI tells me how to go fix these things and you can set a threshold like Engineers can deploy things with a severity of medium or or lower but not high and above that kind of thing so you can set a threshold and say I only want to stop deployments based on high severity issues and then an engineer is gonna go in and fix those So, um, that's Gonna Knock things off as we go through the cluster over time. We also have cost functionality which I mentioned and I again, I can't show you the entire overview of the whole product and in just a few minutes, but I'm giving a real quick view.

So here we have breakdown by cluster across the organization average build cost total build cost. If you take our recommendations what it'll be in this particular situation. We're actually recommending that you spend more money and that's because Fairwinds insights approaches the problem of getting configuration.

Correct? Not just save you money at all costs and we bring things down even at the workload level show you the relative total cost of clusters tell you you know, what's set how we think you should change it in the future. You can click on any one of these workloads.

We collect metrics and show you where those resource requests and limited. Anyways, there's a lot more here covering compliance. There's automation so you can kick things out to a ticketing system and an automated way lots lots more if you're using kubernetes, you can have a whole bunch of disagreements with your CTO your CEO your head of engineering or you can use something like Wins insights put it in place trust it out of the box and get some same defaults without spending forever and ever writing a whole bunch of custom policy.

That's Fairwinds insights. This is five disagreements about kubernetes glad to have you here. We'll see you later.