From Chaos to Clarity: Observability in the World of Microservices and Kubernetes at SKILup Days 2024

Hi, everyone. Welcome to the Skill Up Days, uh, talk on observability. And today we are going to look at our topic from cap chaos to clarity.

So what we're going to talk about in general is observability and how it helps, uh, organizations and how it leverages, uh, productivity in terms of microservice and Kubernetes. Okay, uh, so we have two speakers, uh, for this event. One is, uh, Amit.

Hi Amit. Uh, please introduce yourself. Hey, Faruk.

Um, I'm Amit. I've been working in, uh, development and DevOps space for the more than 13 years, and, uh, love being part of the open source space. Cool.

Thanks. Uh, and, uh, so I'll introduce myself. My name is Faruk Kri.

I've been into DevOps for around 17 years, mainly into, uh, banking and the financial sector, focusing more on, uh, containerization and DevSecOps. Uh, and I like to teach. Uh, that's, that's all, I guess.

Uh, let's move on. Right? So, uh, a quick, uh, overview of what we'll be covering today.

So the first, uh, thing is we look at what exactly is observability, because you hear that word a lot. Uh, so we just try to break it down a little and, uh, look at what observability is, and then we'll talk about why we need observability in, in organizations and how it helps organizations to get business objectives, right? And then we look at the landscape, uh, which is really washed.

And Amit will help us walk through the different tools that are there in the ecosystem and how you can use them to leverage productivity and, uh, troubleshoot more, uh, effectively. And then we'll wrap up with a few keynotes. Okay?

So the first thing that, uh, when you hear observability, it's, it's kind of like an umbrella. It's a big thing. And then you can break it down into three, uh, separate streams.

So the first one is logs, uh, which we look at more detail, uh, later. And then you have metrics, and then you have traces. So think of it like an umbrella, which has three, uh, pillars.

The first is logs, metrics, and then traces. So let's see one at a time and see, you know, how, how this, uh, ties into the whole observability, uh, system. And why we need observability is, um, yeah.

So the, the reason we need observability is the first, uh, reason is easy identification of issues, right? So a lot of us have application systems which are spread across regions. They are really, um, isolated in nature.

So what observability does, it helps us to identify where exactly the issue is originating from. And that in turn helps to resolve issues quicker. Then that's what makes business happy, right?

And, and the other thing that it does is, so once you find out, uh, where the issue is coming from, it also helps us to, uh, find out the root cost. Like if the server is down, why exactly is it down, right? Is there a memory?

Uh, constrained is the storage full. So those kind of things are very, uh, useful when you have observability, otherwise, you're kind of shooting in the dark. And then, uh, the, the last thing is you, if it observability, if you do it right, you can also go ahead one step and find out issues even before they happen because these are generally, uh, there are indicators of issues which you can, uh, uh, find out through effective observability, right?

Cool. Right. So let's look at logs.

Uh, what is logs? So logs is basically, uh, a stream of data or anything that's been generated by a system, an application, or any kind of infrastructure. So it could be a server, it could be a container, it could be a serverless, uh, resource that you're using on the cloud.

So these kind of things are basically, uh, anything that's a resource that's running is capable of generating logs. And generally logs are timestamped and they're sequential in nature, which helps to troubleshoot, like, and correlate. Like, if you go in and check, you can go into a particular timestamp of the log and then check it over there.

Uh, logs also have different, uh, levels of, um, uh, uh, granularity. So there are informational logs, which are just for information. Then there is also errors, and then you also have debug logs, uh, which kind of give you a pretty detailed, uh, logging, which is useful when you are troubleshooting, right?

And so the source is generally the resources that we mentioned, like servers, containers, and, uh, app, uh, operating systems as well. And logs are generally in text format, but you can also have, uh, different formats like JSON and XML also in certain scenarios, right? So logs are, are generally the starting point when you want to get started with observability.

And the, the challenges that you might face with, uh, logging, uh, implementing logging are, uh, threefold. Uh, so the first would be volume. So, uh, it's, it's sometimes very shocking the amount of logs the system can generate.

So you have to kind of plan ahead that what kind of volumes you're looking at for each application or each system. And you have to kind of do that capacity planning if you want to do proper logging. The other is also the speed at which these logs get generated and how do you rotate these logs and how, what do you, uh, you know, how do you manage this, uh, huge resources that are ting logs every second, every minute, and just pushing them out there.

And the third is, uh, the variety, because all systems have a slightly different view of, uh, you know, structuring their logs and their formats might be different. How do you handle this? And then, uh, inspect these logs to make sets out of it, right?

So to, to handle this, you generally have, uh, best practices that, uh, you need to do if you want to get the most value out of your, uh, logging solution. The first would be centralized logging, because the, the last thing that you want to do is, uh, be running around and trying to figure out where your logs are. So the first, the first thing would be to centralize your logs in an central place, and then you can build on top of that and you can do storage optimization and then visualization and all of the other stuff, which adds a lot of value.

The other thing would be having some kind of log rotation. Uh, so it could be based on time. So for example, uh, every day.

Uh, but the recommended one is based on size, uh, where, you know, you have a certain size when your, uh, log file, which is a certain size, uh, maybe one gb, and then you start creating a new log file. So that's the recommended way that you would want to look at, uh, loggings. And the third is, uh, like I mentioned that there are a lot of different type of, uh, loggings that happen.

You have informational logs, which generally are only useful at a point of time, and they really don't have any, uh, retrospective value. Uh, so you kind of want to drop those logs, and there are solutions that help you to do that. So the only logs generally would be then, uh, you know, debug and error logs, and then you have different ways of handling them.

So the way the logs are, uh, are labeled would also, uh, uh, influence the way that you're going to treat these logs and, uh, store them, right? So that's logging in a nutshell. So the next, uh, stream, uh, under observability is metrics.

And metrics are kind of like, um, uh, a, a data point, uh, or, uh, a space of time. So that's how you kind of think about metrics. So for example, what is the performance of this server at this particular time?

Uh, so what's the average for the past 24 hours? So that's, so metrics can then further be broken down into system metrics, which is mostly the hardware or the use resource usage, like the servers or the operating systems. And then you have application metrics also, for example, latency and how, how slow is the page loading.

So those would fall under application metrics. And then last, you also have business metrics. So business metrics would, uh, basically mean that, you know, what is the SLA, how much uptime do we have?

Is the application going down? So these are basically SLAs that the business really needs to adhere to, and these would fall under business metrics. All right?

Right. So the third one is, uh, tracing. And this is generally the, uh, the harder not to crack among the three.

So tracing is something where you try to create a user story where the user comes into your application and then he goes through a certain journey in your application, right? So that's where, you know, uh, you try to that, and what this does, it gives you a lot of information about how his experience was, where he's navigating, what, what, what are the bottlenecks that he faces. So for example, a user app and today's applications are all microservices, and they're all different applications within applications, which is microservices.

Uh, so you want to be able to find out if a user had an issue and you want to be able to do that fast so that you can tell that this is exactly where he faced a slowness in the issue. So was it, uh, a microservice microservice, was it a database or was it just across the application? Why is there a latency, right?

So the, the ability to do that, uh, and uh, look at issues like with through an x-ray lens is what tracing enables you. And this is more towards performance optimization also, right? And, and when you have all this information, what it does, it helps you to correlate these data, right?

Okay. So some of the use cases, like I mentioned, is, uh, why observability is really important in today's time is because you, uh, a a single application would be talking to multiple different resources, multiple different, uh, components within a single application, right? And debugging, this could be a nightmare.

So the first thing is that it helps you whenever you need to debug and find out issues, which is, um, uh, you know, where issues happen, you want to do this. Another thing is you want to monitor transactions, right? So if you're into banking or finance, and you want to understand that this application or this transaction failed, right?

So you can't really afford to have a failed transaction. So you want to be able to highlight and immediately, uh, look into this or look at transactions, which are, you know, anomalies which are outside of the general behavior. Those are things that you know, you can do with, uh, tracing.

And then the third one is failure. So why did the transaction fail? And if it failed?

So there would be alerts that you can configure and, and catch that with tracing. So it gives you the ability to go back, uh, in the journey where the user was and, uh, fix that or monitor that, uh, transaction. Okay?

Right. So now that we know, we went through, uh, what is observability and why we need observability and what are the three general streams that we have under observability, uh, Amit will help us to walk through the different tools that we can use for these, uh, metrics, uh, tracing and, uh, logs over to you. Thanks, Faruk.

That was fantastic insight into observability. Uh, so for now, what we are gonna do is we are gonna walk through Prometheus. Uh, proess is one of the oldest open source projects, which has been used for metrics.

It has more recently been adopted, uh, by CNCF and is guided by the CNCF in order to make it better and stronger and ensure it's always open source. Uh, so one of the main things which promeus is used for is collecting and storing times series data, uh, more specifically metrics from everywhere, whether it's applications, containers, servers, so on and so forth. Uh, all of these metrics are gathered and then finally stored in a specific type of storage.

We then enable it to be queried using ProQ so that we can perform different types of analysis and aggregation. And finally, we are going to use all these metrics in order to alert DevOps engineers or SRE engineers in case of any issues, um, any specific issues that the metrics highlight. And finally, it all comes together with visualization.

So what is the Prometheus use cases? It's all types of system monitoring, whether it is resource usage monitoring, such as CPU memory, uh, you wanna find out the performance and availability, which is like uptime latency of your issue, um, or anomaly and bottleneck detection. But again, it goes back into latency of your application.

This gives you real-time insights into all sorts of metrics that your application ensures to export. So how does all of this get visualized? Uh, you are going to visualize all this metrics and information in a tool called Grafana.

It is second, another open source tool for visualization, which specialize for monitoring metrics logs. Uh, it all comes together in one single GUI where you can visualize whether it's bar graphs and charts, uh, whether it's uptime and downtime. Uh, all of it comes together within the setup, including your logs, which generally people tend to use by leveraging Loki.

So how does a Grafana dashboard look on an average? So on the right, you can see whether you want bar graphs, you want dials, you want numbers, you can get it all. You can pick and choose what is your preference based on your end users, and decide what type of visualization you wanna provide them.

Whether it's monitoring infrastructure, whether it's application performance and latency log analysis, everything can be done your in one single setup. We then go to Yeager. Yeager is an open source distributed tracing tool tool.

So this has been more recent of a tool, uh, in terms of versus Grafana and Prometheus, and this helps us in tracing and troubleshooting transactions in complex microservices architectures. So what does exactly Yeager do? Yeager is going to try and trace a request all the way from the beginning or your first microservice all the way to the backend or your database system.

It helps you to understand exactly how your transaction correlates from microservice one to two, to three, to microservice n. What does this help with and why do we need to do this? This is so that we can visualize dependencies, we can ensure that we can find out which microservices taking the maximum amount of time or in case of failures, exactly which microservice caused this failure.

It helps in planning for better resource optimization or ensuring that each microservices receives whatever resources are required for it. And lastly, but finally, it all comes down to improved reliability, improved monitoring, helping in debugging your systems and improving your SLOs. So all that you want to do finally is ensure that your MTTR is really low.

Your MTTF is really low. And how do you ensure all of this? It's by ensuring that it's very easy to identify issues in case of a failure, and you can resolve it really quickly as well as you can monitor every single microservice so you can identify a microservice, uh, when it starts showing symptoms of failure, not before it actually has a complete outage.

And lastly, and not, uh, it's, it's now come back to ai and we have a new tool called K-H-G-P-T, which has very recently been adopted by CNCF as well. And water has K-H-T-P-D do, it's a operator, which is driven by an AI engine. You can pick and choose your AI engine, whether it's open AI, cloud, so on and so forth in order to manage, troubleshoot your Kubernetes clusters, your logs, et cetera.

So this diagram shows you a very well explained example where it analyzes your clusters, configurations, it processes all your logs, it evaluates all this data and information, which would be more complex for a human to, uh, kind of go through and analyze. A machine, on the other hand, can take all this information inside and process it in a very understandable manner and provide you proper outputs and insights along with recommendations of how you can improve some of your, uh, performance, as well as in case there are certain logs which are constantly throwing errors, it'll provide you recommendations of how you can permanently fix these issues. So as we were saying, it's a single Kubernetes operator that you install within your cluster, and it performs all the respective actions that it needs to do, whether it's understanding the complex problems, diagnosing all the different issues, and then finally going and recommending potential solutions or optimizations.

Right? So Amit, just, uh, I think, uh, with this information, if you look at this, uh, so you have for logging what, what, uh, the tool that I think is the industry standard is Prometheus, right? And, and then you have, uh, for, uh, metrics, uh, sorry.

So, so for metrics, uh, uh, for visualization, you have Grafana and then you have, uh, Yeager, which is for tracing. Is that right? Yes.

And generally for logging, we tend to use Loki, which is again under the Grafana suite. So you, you are absolutely right. So basically the key takeaways as you can see is metrics collection, which is mainly Prometheus visualization, which is mainly Grafana distributed tracing, which can either be Eger, which is, uh, managed by CNCF or Grafana, uh, version of tracing tool and centralized logging, quite a few of us use Loki and AI management is completely new in this whole realm of things where case GPT is now the new tool, which a lot of companies have started leveraging, but they might be a lot more to come.

Yeah, I think it's very interesting because uh, you always had observability for a while and you had these three things, but now there is kind of a merger of AI and then you can leverage AI with observability. So that kind of puts observability on steroids and then you can explore more with, um, power of, you know, chat o ai, uh, adults, right? Absolutely.

So one of the things previously is a lot of proprietary vendors used to sell you the idea that we have specialized, um, intelligence in the background where we can analyze all your logs and provide you these great insights, which would be very difficult for you to analyze. This has now been democratized and it enables a lot of people using open source to, uh, use similar such logic, um, along with which there are even formulas nowadays, which can help with anomaly detection, where you can add that formula into your Grafana dashboard. And what that does is it compares your current week with the past week and checks for deviation from the norm.

And if there is a deviation more than X percentage, it'll send you an alert saying that, hey, something looks different within your systems. Right? Right.

And also with uh, yeah, I think, uh, Kate's GPT, you also have the option not only to use open AI uh, models, but you can also use Claude and any other models that you choose to. Yeah, You can even use a model you choose to deploy on your own. So it also supports your, um, lama, if I'm not mistaken, which is currently, you can set it up yourself on your local machine or on your respective server and run it by yourself if you want your data to stay with yourself, and you do not wanna send it anywhere.

Right, right. Yeah. Makes sense.

All right, awesome. Thanks Faruk. That was a great session.

Yeah, thank you. Thanks a lot, ameds, fair time. I think this was good.

And generally when you want to start off with observability, I think, which one, which part do you recommend the most? Like, uh, logging metrics and traceability, how do you recommend, uh, somebody just getting started should get into this? I would probably, uh, jump between metrics and logging.

So I find that you need metrics or tools such as Prometheus in order to understand anything that you have. And the second thing is logging. You want one single location where you can read all your logs, understand what's going on in case of any failures, so on and so forth.

Yeah. Yeah, I think, yeah, that's generally the way to go about it because I think tracing is generally the, the second phase where you want to get more mature in jobs ability then start with tracing, uh, because it takes a little more hands-on uh, getting and implementing tracing. But you have a lot of tools that are making it easier now, right?

Like open telemetry. Uh, so if, if your views you, you have used a lot of it, right? Faruk?

Yes. I think Uber Telemetry is kind of creating a standard in the observability space where, you know, you need to have a particular standard and then it kind of abstracts that implementation, um, uh, nightmare, which we had for different vendors. And then you can use any, any solution that you want.

But in terms of transmitting the data and filtering the data and order doing all of that, that that is where Open Telemetry is for focusing itself on, which is great. That is fantastic. Alright, so I think that's, uh, that's all, uh, we had.

Anything else you want to add, Amit? No, I, I think we have covered it all on a decently high level where, uh, this topic would be good to discuss between many different peoples on intermediate to high level to, uh, it won't be overwhelming for them. True, true.

Yeah. Thanks. It was lovely, uh, talking to Yamit.

Thanks. And, uh, thanks everyone.