Deploy Apache Kafka on Kubernetes in a Time Crunch | DevOps Connect: DevSecOps 2023

Hi, welcome to today today's talk about deploying Apache Kafka on Crunch. I'm, and I'm a member of the Cisco Tech and Incubation group. Let's get started with this.

Your boss or another team member comes up to you and says, I really need a new cluster on our Kubernetes environment, and I need this by tomorrow. And especially if you haven't done this before, then the Googling starts. And some of the things you come across during your research are questions like these, should I use a stateful set or a regular Kubernetes deployment?

What about my volumes? Because I need to store some data. How do I make sure that if a broker goes down, I needs to restart, doesn't lose any data, and then I also learned I need to use a headless service.

How do I go about configuring a headless service? Because a producer and consumer might need to connect to a specific broker. And as any tech practitioner besides these very specific questions, we're also wondering about what are the best practices to solve this task?

And then also, how do we make this reproducible? Because another requirement that only came up later in that discussion is we don't need just one Catholic cluster, but five of them because it turns out the other team wanted one, two. And also we need a different one for, uh, testing changes before they go into production.

And then we need our production, uh, cluster as well. So whatever approach we come up with just needs to be reproducible. And with all of these things in mind, it's feeling really overwhelming.

And it feels very much similar to sitting in front of a, sitting in front of a giant Lego bag or Lego bin where you have lots of different pieces in front of you and you don't really know what piece to start with. Should I start with the, the big pieces, put them together, or should I start with the small ones and then work my way up to the big pieces? And so this just gets really overwhelming quickly, and therefore, let's get some help.

Let's figure out how we can make our lives easier. And also besides the other, looking at the other requirements, most importantly, meet our deadlines because we want to make sure that we're not blocking other people's work with this kind of task, even if we haven't done this before. On top of that, as we just talked about, this diploma needs to be, or this approach needs to be reproducible so we can roll out more clusters over time whenever a team needs a new one.

And last but not least, we want to make this as easy as possible so that other team members can also do all of these tasks without having to go through extensive training or research as well. So the tool we'll be using today to get to that point where we can roll out new Kafka classes on Combinators easily is K operator. K operator is a kaf, uh, is a Kubernetes operator for Kafka.

And when we look under the hood, what that does is really like, there's two main things to it. First of all, it's a Kubernete software extension. So this is another part that runs within our cluster.

But the reason why we have this up and running in the first place is it helps us to deploy and manage stateful and more complex workloads. Because when you think about deploying a stateful workload in general without any additional tooling or help, there's a lot of knowledge and expertise that goes into achieving the task. And if you all do this manually, then if you are not available, then other team members might not be able to do this at all, or they might be able to do some of it, but then they might be missing a crucial step of it.

And therefore, we're using this operator to basically codify that knowledge to make sure it's accessible to everybody on the team. More specifically, what that looks like under the hood is where we deploy any kind of application, but specifically in this case, a stateful application into creditors. This means we're running a lot of a lot of younger configuration.

Part of that is a deployment configuration. We need to configure services. Uh, we mentioned volumes before that have to come in there in one way or another, plus a lot of different, a lot of other different, uh, resources that, that are not part of this slide right now.

But when we look at it, especially under the aspect that we need to roll out more than one cluster, for instance, a lot of these, these, uh, general configurations stay the same. Some of them might change in terms of we roll out more brokers than, than in the other environment, but everything else stays the same, more or less. And what we really want to focus on instead of dealing with these repetitive details again and again, we want to make sure that we deploy clusters that are meeting the requirements of the teams and our end users.

And so when we look at the actual configuration for a Kubernetes operator, or in this case the Kafka operator, so this is a screenshot from the, from one of the example s and the key operator, uh, repository links to all of this are at the end of the slide deck. Um, what we're looking at here is we only focus on configuring things like the zookeeper address or should, excuse me, should clients be allowed to how to create topics, yes or no, or would we allow any, uh, unencrypted connections. So these are all questions or these are all aspects that we need to configure and what we need to put in effort.

And that's where K operator can help us so that we can focus on solving business problems and meeting business requirements and not deal with repetitive tasks so much. And so as we're moving towards the deployment aspects of this, um, before we talk about these specific steps, one of the technologies and tools we'll be using in this process is he hem helps us to bundle everything up into small packages that we can apply again and again so that the actual steps of, for instance, creating a new topic in a cluster is all defined in the HEL chart. And we just have to fill in the blanks, uh, in terms of like topic names or any additional configuration that we might have.

Um, the next step for that is to use, uh, it to a continuous delivery tool like Argo City. And the main purpose of this, if you haven't heard about Argo CD before, is to automate an abstractive way. Our cheap cut apply commands that we would run in the command line otherwise.

And last but not least is not so much of a tool, but more of a process or a way of working. And then is GI ops. And GI ops is our way of defining infrastructure as code and treating our gift repository that contains all the he charts as the single source of truth.

So whenever we need to create a new cluster, create a new topic, or make changes to a topic or whatever, whatever else we might do, everything starts with a commit to the GIT repository and then we can roll out the changes based from that or from that GI repository down the road. The advantage we have from that approach is that every change is traceable and we can audit in case something breaks, we can always roll back to a previous version and we can understand exactly what change caused trouble in the first place, so we can avoid that in the future. Now we come to the specific deployment steps to get to a reproducible credible reproducible Kafka Casta deployment.

Um, I will gloss over the, well, like we'll go over the specific steps very quickly, but at the end of the SL tech, I have a link to a block post that walks you through everything we talked about step by step. Step one is turning all the instructions or like all the steps we need to roll out zookeeper and key operator into your respective head charts. These land charts then allow us to again and again roll out clusters, uh, as we need to.

And then in the next step, we'll install Argo City in Kubernetes. And then once Argo City is up and running, everything is configured properly. We can start by creating the respective applications within ar cd.

And what that means is that from now on, Argo CD knows about the respective hand charts and our GI repository. And so whenever somebody starts to make changes to these charts, then we can start deploying our new changes. And to summarize what we learned or what we talked about here is the essence of it is to automate these deployments as much as possible.

Even when we're dealing with a state full service like Kafka, that is much more complex to deploy than a regular web application, for instance. Um, but we're using a tool like K operator and Kubernetes operator that helps us to abstract away a lot of the deploy details for state stateful or more complex workload. And whenever we make changes to a specific environment, everything is tracked in Git so that we can always roll back and understand all the changes that have been made up until this point.

That concludes today's talk. Thank you all for attending. And on the last slide, as promised, I have a bunch of links to the K operator repository to which is another one of our products that uses K operator under the hood and enhances it with a few other capabilities, and then a blog post that gives you a step by step tutorial on everything we talked about today and links to OL profile.

Thank you all for attending.