A Platform Engineer’s Guide to Kubernetes | Cloud Native Now 2023

So welcome to my talk on a Platform Engineer's Guide to Kubernetes. So I'm here to talk about how if you can enable your developers to deliver apps and services faster, everybody will win. So bit about me.

I'm Danielle Cook, I'm a VP at fairwinds, and I'm also co-chair of the C N C F CARDIOGRAPH Working Group. Uh, I'm gonna give a little plug to my working group right now. 0 version.

Uh, the model assumes you've decided to go cloud native and it takes you through five stages of maturity from free production environments to how you're gonna optimize your environment. So, uh, I just wanted to call this out. io, uh, cause it's a great resource and, um, gives you a bit of background about me and the content I've put together.

Um, and I am gonna just say, you know, I am from Fairwinds, so we do provide software for platform engineers to standardize Kubernetes so that developers can ship applications faster. Now, moving on. Uh, so why we're here today?

So Gartner anticipates that platform teams by 2026 will provide reusable services for app delivery. So the analyst, Paul Deloy, he talks about how platform engineering is, is emerged, or has emerged in response to complexity of modern software architectures. So we know that that's true, and we know that's true with a cloud native environment.

Um, one thing he did say, uh, and I I kind of quote that there are non-experts and end users are being asked to operate in assembly of complicated arcane services. So we are seeing that in Kubernetes. So platform engineering teams, they're using Kubernetes.

They want it to be the customized path they always wanted. Um, and then they're giving it to development teams to use. The development teams aren't always Kubernetes experts.

So you have a challenge with how to manage this with teams of people who need to use the platform, but are also not experts. So some of the challenges we see, um, includes like a lack of visibility into Kubernetes clusters, what activities happening, how it's growing. Uh, it can be difficult to manage and troubleshoot multiple software versions or the helm charts, the APIs, all of that across your organization and your users.

You need to define user roles, responsibilities, and privileges. And then you have to track these across multiple teams environments. You need to then identify role violations.

It's challenging to perform compliance checks and then prove that compliance in this dynamic environment and it's just time consuming. And then finally, like, do you know your cluster health? So many platform teams, like they know they have a bunch of containers out there and a bunch of clusters, but they don't know if it's all configured to meet best practices around performance of your app, the security, your cost efficiency.

So when that's happening, the platform team becomes responsible for everything. You have to set it up, monitor it. You are checking everything from like, is it configured for cloud costs?

We checked all the boxes for security. Are we gonna face downtime? What's our reliability?

The goal of the platform team is to allow many users to develop and deploy across many different Kubernetes clusters. But as more teams do this in production environment, it becomes in just increasingly difficult. You have platform engineering teams trying to manually write or review each dock or file in Kubernetes s and then they're deploying it to class.

It's just, it's a lot. What you end up with is you get this windy road with no guardrails and basically every opportunity for something to go wrong. So you could have, you might be running face images with security vulnerabilities or you might not be aware.

Um, you might run into some, um, kills because you didn't set liveness and readiness probes, right? And you might the platform engineering person know how to do this, but your developers might not. So what you want is this straight road with guardrails that supports developers in their goals, their business requirement to ship applications faster.

And you want to stop serving as this Kubernetes help desk. So what we are gonna focus on today is applying governance and policy, and this sometimes called guardrails. So, um, in the CTO summit with the c ncf F that I attended, and then in various different areas, people are calling a paved road, a golden road or an internal developer platform.

You need to set this up and include the guardrails so that you are creating a self-service model. Your platform engineer is freed up. You get to focus on the sprints and the innovative activity you wanna do.

And your developers, you know, they're doing things right. So Kubernetes governance, it helps ensure that there are established processes, policies, so that you can implement and run Kubernetes successfully. It ensures that complex environments, uh, that in the complex environment you're enabling your developers.

You can meet the organization's policy requirements, you can adhere to best practices and also like meet regulatory requirements that your industry might face. So with governance, you kind, you get an enterprise grade production ready, Kubernetes environment at scale. Whew, awful.

Um, and you have a defined way to manage and maintain Kubernetes and a process for managing security bug fixes and all the configurations that developers face when using Kubernetes. It kind of sounds blah, right? So governance and policy, it's not like the most exciting term you might have heard all day, but it's needed.

You need to standardize and use guardrails so that you can use Kubernetes in the enterprise and at scale. So kind of defined here, it's a strong fence. So platform teams need this.

Like it's not a nice to have its requirement. Um, so if the one thing that you take away from attending this session today, it's like you, you need to be looking into governance, you need to be looking at it. Um, so how do you actually do this?

So you need to create Kubernetes guardrails and then you need to automate them and make sure they're enforceable. It's no good like writing something down on paper and hoping for the best. Like you need to make sure there's a process and it's built into your development life cycle.

So there are kind of, in Kubernetes policies are obviously defined in code. And so policy is code gives you lots of benefits, version control, auditing, testing and repeatability. And we see these falling into three main categories.

So you have your standard policies, and this is, I'm gonna enable some best practices across the organization, my team, my clusters. So some examples like include disallowing resources in the default namespace requiring resource limits to be set, preventing workloads from running as group. Then you have organization specific policies, and this is where you're gonna enforce best practices that are specific to your organization.

Um, and they're often requires some customization. So some examples include requiring particular labels on workloads, enforcing a list of allowed image registries or policies, um, that can help you with compliance and auditing. Um, and then you have environment specific policies.

So this is where you wanna enforce or even relaxed policies for particular clusters or name spaces. So some examples here are like stricter security enforcement in fraud clusters or looser enforcement in a namespace that runs low level infrastructure. Now some examples or like, you know, I I mentioned earlier like putting 'em on a piece paper won't work.

You need to automate these. And so some more examples that I wanted to dig into. We bucket them into kind of three main categories, security, reliability, and efficiency.

So for security forests, you have Gabriel Rails to prevent containers running as root or avoid allowing privileged escalation. These are problems that continuously come up in clusters. Um, you might be surprised or maybe you're experiencing it.

Um, for reliability, you'll want readiness and liveness probes in place. So readiness probes enable your developers to instruct Kubernetes that this container running should not receive traffic until additional tasks are completed. So just loading files, for example, um, liveness probes evaluate what they're in an application that is running in a container as healthy state.

Um, if not Bernet kill the container attempts to redeploy. Like you want to set these for efficiency, you need to set guardrails on how developers can set CPU U and memory limit. So oftentimes the default is to not set any of these, which could mean you get a ridiculously high cloud bill at the end of the month.

Um, but if you set it too low, it means your app could die and that's not good either. Um, so I did include a link here. We do have an open source project called Polaris, which is an open source policy engine and it checks a bunch a number of development best practices.

Um, so if you're interested in seeing all the best practices that Polaris checks for, you can look at the link flares dot doc, fairwinds, um, and we'll go through some of those best practices. But this takes me onto, now how do I implement these guardrails? The good news is bunch of open source out there.

So obviously as a Kubernetes user, you are going to be a big fan of open source. Um, Fairwinds has a few of these that I mentioned here, the Polaris, like I just talked about, that will give you development best practices. Uh, we also have Goldilocks, which helps with CPU and memory recommendations because oftentimes the reason developers don't set 'em is because they don't know how to set up, like they don't know at what point to put it at.

And then we have Nova and Pluto, which can find outdated or deprecated help charts. Um, now we also have customized like, well there's the customized policy. So there's opa, which which is the o open policy agent, is the C N C F project.

Um, so you can use that to create custom policies. You'll have to do that in rego, um, which is a development language for those of you who are interested. Um, so you'll need to learn that and put those together.

And then if you are doing it, if you are looking to set up guardrails in multi cluster environments with multiple teams, you are at an enterprise, you have a lot happening. That's where something like Fairwinds Insights can help you implement guardrails and also enforce them. But what's really important here is that you implement these policies, you monitor them and you enforce them.

So where do you need to do that? So it's really everywhere from your pull request all the way through production. So every time code is committed, it should be checked in the CICD process.

You should have an emission controller set up so that it's scanning every time there's an deployment against all of your development practices. And then of course, like you need runtime scanning. So if the new vulnerability is announced, you wanna easily be able to scan your environment, whether that's you're creating a custom policy in opa or you have something that is scanning it automatically.

You need to be able to find that. So as you're developing your development life cycle, you need to make sure you're scanning everywhere. So that's huge rapid fire of why every platform engineer needs guardrails and Kubernetes.

Um, you know, guardrails or policies or governance, all of the kind of language is interchangeable. Um, but you need standardization if you wanna use Kubernetes at scale. Um, but, and it's, it doesn't just end with the platform team, it's for everyone in the organization.

So digging into this a little bit more, so across your organization, when you put standardization in place, everyone benefits. So if we look at the platform team, you make it easier for developers to deploy applications, which means it frees up your time significantly. You get to swap like the manual checklist process, which we've seen many people do, where every time there's a code commit, you're checking through the list.

Um, and instead you're using automation so that the developer knows what to fix before it even gets to you. Um, and this does enable development to ship faster because they're going to be learning on the job, the things they need to check and maintain. It gives developers relief.

So again, when I was, you know, with a group of CTOs in a room, they were all saying, everyone is asking developers to do more. We want them to be experts and this and this and this and this, and it's just a lot and they need some relief. So when you use gover governance guardrails policy and enforce it and automate it, you're giving them some time to like just focus on the app.

Let them be the experts in the app. Let the Kubernetes expertise be automatic. By doing this, you can increase their development velocity.

It makes it easier for them to use Kubernetes. So you'll have higher adoption of the platform that you've worked hard to put in place. And you should, when you're doing this, provide them automated ways to fix their code.

Like make a pull request, it gets fixed. You should give them advice if code failed on this is what needs to happen. Um, but you should do it in the tools they already use.

Like don't ask them to go use another tool. If they're using Jira, let them use Jira and give them the governance and the policy and the guardrails there so they understand what's happening in their usual workflows. Moving on your compliance and security teams will be happy.

So these teams need evidence, like your compliance teams needs it. So if you're using this, you'll have your policies in place, you'll be able to show where you're enforcing them if there was a violation and how quickly it was fixed and where it was fixed. For security, again, we've, we've talked about your developers aren't security experts and Kubernetes is not secured by default.

There's a lot of good tools in it, but it has to be configured correctly. So if you are using implementing these policies in this government, you can then automate that and security has peace of mind. You're giving them the checklist.

And then finally for finops teams or your cfo, they're gonna be happy because you can prove where you saved money. So whether it's showing, hey, this is how much cloud spend cloud resource we were using and we made our clusters more efficient and we saved this. Or you'll be able to demonstrate where you might need more memory or C P U because you can show them this is what we were using, we face them downtime, some reliability issues, so we made it greater.

And you'll have demonstrate demonstrateable evidence as to that. So kinda to wrap up in this rapid fire, um, is every platform engineering team they need to implement Kubernetes governance, guardrails, or policy. Like you have to do it.

Don't be the team that gets in the way of development. Be the team that provides developers with a platform that they can use to ship applications faster. So that's my talk around it.

Thank you all for attending. Um, if you have any questions, feel free to reach out. com.