Common Problems with Kubernetes Continuous Delivery at Scale at Cloud Native Now 2024

Thank you very much everyone for watching this session. My name is Bob Walker. I'm a field CTO at Octopus Deploy, and I'm really excited to be here today to talk to you about common problems with Kubernetes continuous delivery at scale In terms of an agenda for this particular video.

First we're gonna start off with a common Kubernetes deployment pipeline, just so everyone's kind of on the same page, you know what we're talking about, and we'll, I start identifying some of those challenges. Then we're gonna move on to some deployment scalability problems once you move beyond that. And then finally, we're gonna wrap it up with how Octopus Deploy can help you solve some of those challenges and problems.

So let's first start off with a common Kubernetes deployment pipeline, and we're gonna start at the very end of the pipeline, which is we want to get the latest version of our code up to Kubernetes. Now to do that, we're gonna be updating our manifest files, or it could be a helm chart or a customized file, whatever you prefer. And this actually highlights one of the very first challenges of a common deployment pipeline, which is what's gonna be the utility that runs QCTL apply that runs helm upgrade or helm install whatever tool you prefer.

And so we kind of end up in this situation even at the beginning of what's responsible for this, but let's proceed along with our pipeline. It's important to note that all we're doing is we're just telling Kubernetes the version of the code that we want to run. Kubernetes still needs to go out to our container registry and download that version of the container and start running it.

But then when we need to update to a new version of a container, this is where our build server comes in. And this is like any other build server that you're used to using. Could be running Jenkins, GitHub, actions, team City, Azure, DevOps, whatever the case may be.

It's modern in your source code. If it detects a new change, then it's gonna go ahead and build it, create the image, perform the tests, any analysis, all that other good stuff that happens with your verification. 1.

This highlights the second challenge that many people encounter with their Kubernetes deployment pipeline, which is what is gonna be responsible for updating the manifest file to then give to Kubernetes. Now, typically, we see companies approach this from a variety of different angles. We've seen all requests, GitHub actions, Jenkins jobs, anything in between.

But really the ultimate goal is to get that version down to the manifest files so we can then send it to Kubernetes. So really, we have two big questions to answer. Now, one of the things that you might be thinking about is, well, isn't this what GI Ops is designed to solve?

Well, yes and no. When we look at GI ops and we look at the four common principles of GitHubs, we can see that it's really more focused on how do we get a file into source control, and then once it's in source control, how can we get that to our desired system? And then how can we make sure that that desired system matches what we have in source control?

Ultimately, what this is designed to do is if we keep an eye on the bottom left hand part of the screen, we're changing this problem from a push where we're trying to push our changes from Kubernetes, from Git to Kubernetes, to moving it to Argo, where a tool like Argo can then monitor our Git repositories and pull any appropriate changes in, but we're still not solving the core problems. So let's start talking about some deployment scalability problems, because as folks start solving some of these challenges, they're gonna start seeing these problems, but not really until you get to the point where you have dozens, if not hundreds of applications. First up, let's address the elephant in the room, which is GI Ops does solve a problem, but it's really focused on solving, getting the update and manifest files to Kubernetes.

Something still needs to update the manifest files and push them up to Kubernetes. On top of that, chances are you're not gonna have a situation where you are gonna make a change to your source code and then immediately push it up to production. Most likely, you're gonna have some sort of a series of environments like a dev test, staging and production.

And so we need to progress those changes through there. But GI Ops and conversely, Argo, they're monitoring just a single file. Once that file changes, then we automatically apply it to say, our Kubernetes cluster.

So what this will look like is we have our manifest files. We'll have a say, a configuration file for our first environment, and then we'll have additional configuration files for our remaining environments. In this particular case, I'm using customized with overlays.

Then if we're using, say, a tool like Argo, or even if we're using Jenkins or GitHub actions, it's monitoring for that development config to change, and then it automatically will sync that to Kubernetes, and we'll repeat the same for each of the environments that we have. Then when we want to have a new version, what we have to do is we have to figure out, okay, we still need to update that development config file. And then once that's done and it's saved into version control, then the applications will start and I'll say, oh, I see a new version.

I'm gonna go ahead and sync that to Kubernetes. And then we'd have to re repeat that by updating our test config file. And this could be done in a completely different way.

Maybe development is updated by a build server, whereas test that's more of a manual process, or it's poor request, same core problem where not really problem, but it's more along the lines of we just apply that change to Kubernetes, and then we repeat that for each of our environments. But then when we start digging a little bit deeper into this, some problems start to manifest themselves. First up, if you have the capability to approve a pull request and for a production config file, that means you can make changes directly into production.

That's pretty scary. How can you ensure that everyone has the appropriate controls in place and that all the approval pipelines have been followed? When you start changing up how you make these changes to your manifest files, that's scary as well.

Having your build server update, your development config file, but then having a completely different process for tests and then staging in the production. In addition to that, how do we handle anything along the lines of, say, environmental differences and secrets? Now, for some differences, we can store them directly in, say, our customized files or our manifest files or our helm charts, but there's still those secret values that we have to be concerned with.

And then finally, what about the different versions? How do we know what version is, is in what environment and is ready to test? These are all the different challenges that start manifesting themselves with a pipeline like this.

Once we start adding in environmental progression, the concern that we have is that when we start solving these problems, oftentimes it's the responsibility of the application team, and they'll come up with different solutions, and that's not so bad for one or two applications. But then imagine trying to solve this problem for 30 applications or even hundreds of applications on top of that, it's common for HA to have multiple different hosts for each of your components. So your application itself, it might run in Kubernetes, but you might have a database backend that's running on Azure SQL or A-W-S-R-D-S, or you might be leveraging file storage and you use an Azure file storage or AWS S3 or anything in between.

On top of that, how do we debug these applications now that they're running in Kubernetes, especially if we're moving off of, say, a traditional Windows or Linux application host where we're used to SSHN or RDPN into the application. So we start compounding additional problems. What's gonna be responsible for doing those deployments to Azure sql, and how do we debug our applications?

So let's talk a little bit about how Octopus Deploy can help solve these challenges and these problems. So when we take all of the different challenges and we start applying them to our pipeline, we've added environments. We've now, we have additional cloud infrastructure where maybe we're hosting our database on RDS.

We have our backend system and files stored on S3, and then we're using Route 53 for any of our DNS configurations. So you can see all of these different challenges that are just compounding themselves on top of one another. So Octopus Deploy fits into this equation is we, we will sit between your Git repository and your container registry, and we are aware of all the different clusters that you want to deploy to, along with any additional infrastructure.

We also have concepts such as dev test and prod concepts of environments. 1 to Dev, what we would do is we would create a release, and this would pull the manifest files, the Terraform files, as well as the version from our container registry, and we would create this artifact that we could then deploy to dev. We can also have that artifact also be used to deploy to our cloud infrastructure.

But then when it comes time to promote it up to test, we can go ahead and push the button to promote that change, exact same processes before. And then when it comes time to deploy to production, then we can start leveraging additional things like our RBA controls where we can ensure that the person who's deploying up to production has permissions to do that. Not only that, we can ensure that your policies are being followed by integrating with say, ServiceNow and creating a change request that then has to go through its own approval pipeline.

We also have features such as our runbooks, which allow for common day two operations. So let's take a brief teaser look at what this looks like with Octopus Deploy. So this is my application dashboard, and I can see what versions have gone out to my development environment.

But you can see that these can't be promoted to test staging or production because these are changes that haven't been approved yet, but changes that have been approved and that are in main, that can go to test staging. And then production. I have a change ready to go to staging, so I'm gonna go ahead and click on the Deploy button, and I have the option to deploy now or later.

I'm gonna pick now because this has to be pretty fast demo. And so then what's gonna happen is now we're gonna get a list of all of the steps that are gonna run for this particular deployment. It's more than just deploying to Kubernetes steps seven and step eight.

That's what's doing the deployments to Kubernetes. But we also have additional steps where we're gonna verify the deployment, perhaps update a load balancer, as well as perform any sort of notification. We're deploying into our database changes by building out our Delta report.

We're also gonna pause the database, the deployment, excuse me, pause the deployment and notify our DBAs and allow them to review the migration scripts before they go out to our staging environment. On top of that, we can look at the history and we can see who did what, when, where, and why. You can see I was the one who triggered this, and we can also see the date in which this was triggered.

Now we're able to do all of this by looking at our deployment process, and what we can see here is that we have a single deployment process that's defined for all of our environments, but what we can do is we can turn on and off steps based on the environment that we're going to. So for example, steps four and five, we're only gonna run that inside of staging, but we get a lot of the same benefits of what we do with, say, a GI ops based tool, because we are pulling our manifest files directly from Git. So we're following a lot of the same principles of GI Ops.

We're storing our manifest files inside of Git, and we're automatically pulling, pulling those, and we're applying that to the system. One of the reasons we're able to get away with having a single deployment process for all of our different environments is by leveraging what we call project variables, where we can scope different values to different environments. In addition to that, we can have sensitive variables, which will only be decrypted when we are performing an actual deployment itself.

So if we go back to our deployment and we can see now, it should be in a paused state, and it's waiting for me to approve it. We can see the database change. I can go ahead and download the file, and I could see, oh, this particular change is pretty innocuous.

If there was something in there that looked to be damaging, I could go ahead and abort the change. But in this particular case, I'm gonna go ahead and proceed, and then it's gonna do my database deployment, and then it's gonna do my Kubernetes deployments. And we're able to deploy to Kubernetes because we've installed what's known as an agent on the Kubernetes cluster.

That's how we're communicating back and forth with the Kubernetes cluster. And then finally, we can actually see the status of our Kubernetes deployment in near real time as it's performing that. So if we wait a couple seconds, we can actually see that occur, jump back to the task log and see where we're at.

We can see where we're at. The Deploy database changes, and now we are creating the connection string secret. So now we can take a look at the Kubernetes object status, and we can see that it is finished, that up and everything looks good, and then it can move on to step eight.

We can see all of the different work that the Kubernetes cluster is performing. So that was a very brief demo into what Octopus Deploy can do. You might be asking, what about if you're using Argo cd?

Well, the good news news is is that we recently acquired codefresh. And codefresh brings a lot of the same core concepts such as environmental progression and environments, and having arba RAC based controls to Argo cd. And so you get a choice.

You if you wanna use Argo cd, you can use Codefresh. If you wanna use Octopus Deploy, you can use Octopus Deploy to D, deploy directly to Kubernetes. Finally, if you'd like to know even more, we have a white paper that you can download for free that talks about a lot of these same common problems, as well as goes into uh, deeper dives into some other topics.

com. Thank you very much.