Welcome & Signals That Save: Using Metrics to Predict and Prevent Outages – SKILup Days 2024

Hi, um, welcome everyone. We are so glad that you've joined today. Uh, so we have, uh, the theme of the day's observability, and we have 10 interesting sessions by industry experts lined up.

And then we are very thrilled to kick off this series of sessions, uh, on a very critical topic, very crucial topic, which is observability again. So with the first session, what we're trying to do is, um, we are gonna briefly describe what observability is and how, uh, how important it is in the current software industry and, um, um, why it's very crucial to, to stay ahead of failures and keep the system running smoothly. So that's what you're trying to cover in this first session.

The title of the topic is Signals That Safe Using Metrics to Predict and Prevent Outreach. Um, little bit, a bit about us, and, um, so, so there is, let me introduce ourself first. I'm Vinod, um, and I'm joined by my colleague and a good friend, Santo.

So we both are cloud and DevOps engineers. And, uh, we have a passion for platform engineering. And, uh, we're apart from the technical bits, we are very passionate about, you know, uh, we are very active in the community, uh, in, in the open, open source and, uh, cloud native space.

And, uh, OSH is a C speaker in, um, and has shared his insights in multiple international conferences. And we have been organizing events, advocating, uh, the cloud native best practices and also innovation. So that's bit about us.

Um, so I, I'll, I'll start off with the saying, um, which, which goes like this, so you can't improve what you can't measure. So this specific saying lies in perfectly with the topic today, which is observability. So we need to measure key metrics like, uh, traces, uh, metrics, uh, traces and logs to gain the visibility that are needed to detect and prevent issues.

Uh, so, so that it helps us to understand, um, the problems that might occur. And you can even prevent issues even before it occurs. So this, this saying goes, uh, very apt with the topic today.

Okay, so straight to the point. What is observability? So we had this traditional monitoring, um, which has served the purpose for so long.

When we had this, um, um, monolithic systems, traditional monitoring was sufficient. So to understand what if, if a fault occurred or if a problem occurred, to understand what is a problem, and to diagnose the issue, the traditional monitoring was good. So in monolithic system, what has happened is, like you have the entire applications run in one system, and then all your logs metrics are all at a centralized place.

So to diagnose and to, uh, find out the problem, it was easy. But in the, in the current times, we have more distributed and, um, uh, decentralized and distributed and microservices and serverless, uh, kind of systems, um, more in use. So the traditional monitoring doesn't serve that purpose are currently, so you'll have to have an upgrade to the traditional monitoring.

So that is observability. So if I have to give a definition for an observability, it is, it is like an upgrade to the monitoring and, and it understanding the internal state of the system based on the outputs of the system. So you'll have metrics, logs, and uh, traces.

And based on that, you need to understand what's happening in the system. So when you have multiple microservices, they're all interconnected in a complicated way. And, and to understand how the entire system works, you need to have the entire context.

You, you, you can. So this individual microservice, again, will create its own metrics and alert metrics and, um, its own logs and its own traces. So to have a holistic view of the system, you need the entire thing.

So that's observability, uh, for you. So it provides, so again, going back to traditional monitoring, it was wherein you will define some, uh, predefined thresholds like CPU, uh, or memory thresholds. And once it crosses that limit, you will be alerted and you'll either do some manual activity or you do some automation.

So that's how traditional monitoring used to work. But in the current days, um, observability, um, in a microservices space and distributed kind of, uh, setup, you need something more proactive. Um, and, um, so you need to understand the unknown issues.

You can't always go with a predefined threshold or you need more kind of proactive setup to cover all the basis. So that's observability. And, um, if I go to the next, and, and I think it sort of covered how does it matter?

So, um, if you look at it, um, firstly the increased complexity of the software system. So now we use, uh, microservices containers, serverless architectures, and, uh, without the complete visibility in the context, it'll be very harder to identify and fix problems and liken the monolithic, uh, architecture. And also these systems are very dynamic in nature.

So, um, it, it gets autoscale based on the demand and, um, um, it, it moves the workloads between systems all based on the demand and the requirements. So to manage this sort of dynamic environment, you can't have a static monitoring, which is not sufficient. So that's why observability, uh, is very crucial.

And, um, if you look at the other, uh, big reason why it matters more, it's like, it, it of course prevents the downtime. So, um, we know what out pages or downtime means, so it, it is lost revenue. It is, uh, lost trust for a, for an application or, or an enterprise.

So observability helps you identify the issue proactively and prevent failures. So it, it identifies even, uh, even when in a, a problem request. So you will, you'll be ready.

So, um, you can typically identify the problem. So, uh, if you look at it, it improves the reliability, right? So, um, it, it helps in reducing the MTTD, which is meantime to detect.

So the pro, the time it takes to detect a problem, but the time it takes to resolve a problem, uh, an outage or an issue has drastically brought down when observed was introduced. So that's why it is, um, it is very crucial in the current times. So that's about the criticality of the, um, the observ, if you have sort of figured what is observability and why is it crucial.

So now onto the, um, uh, the key pillars of observability. So if we have, uh, three key pillars in observability, which is metrics, logs, and traces, and, uh, if I have to briefly tell you what each does. So metrics is like, uh, the numerical data points that represents the state of the system.

So for instance, if I give you an analogy, if you're driving a car, you're seeing the speedometer, and that's a metric, right? So the speed at which you're going, so that's a metric, a specific, it's a data point at a specific time. So similarly in an, in a software system, you'll have metrics like, uh, CPU usage, uh, memory usage, or, um, so similar to that, Eric.

So it's, it's a, that's a metric. And then when you come to logs, logs are more like, um, it's, it's, it's a detailed timestamped record of events. So, um, it's like your system events in the system, or if there are any error message in the system that gets logged.

So that's the log, right? So if I, again, give an analogy, uh, think of like a, um, log book for a driver. So if he keeps the log book, and he records when the trip started and when the passenger started with the timestamp, and it's like a log, right?

So he recording all the information with the detailed timestamp and information. So similarly in it, the software system, you'll have logs that's collected. And then the last one is traces, which is like, um, the com it, it is recording the complete journey of a request as it travels through various components, we have explained, we have, we sort of, uh, um, uh, pointed that in the recent, in the current systems, we will have microservices and, um, they're interconnected.

And a user request might go through different microservices, and it might involve, uh, calls from one microservice to another. There'll be API calls, stuff like that. So you, you need to have the entire trace.

So how, how the entire system, entire request have gone through. And, uh, how does the, uh, the request app, uh, gone from one to another. So, because this is very crucial in the modern distributed systems, because only if you know, um, uh, how the entire, uh, request app processed through different systems, you'll understand how, how it, uh, worked or where it failed.

So there could be a problem where wherein, one, the request has failed and one specific microservice, and if, if we have no proper trace, it'll be very difficult to understand where it failed. So metrics, logs, and traces very important. That's three key pillars of observability.

And if I go to the next slide, um, this are the, um, this is from Google's, um, reference. So Google in their site, reliability engineering, um, says these are the golden signals for observability. They listed four golden, um, um, signals, as they call.

These are telemetry signals, uh, for any user based systems, any user, uh, um, like any systems that user being accessed by users like a public open user systems. So they are latency traffic errors, and saturation latency is like the delaying system. So imagine if you're accessing a website and, um, um, the web webpage is taking time to load.

So you need to understand what's the latency. So if you track the latency, you'll understand, uh, where it is being, um, slowed down, and what is the latency time, um, where is the de query? And the second one is traffic.

That's the number of transactions process per second, which means the amount of load or what is the demand for the system, how loaded the system is. So these also need to be tracked to understand how the system is performing. And the third one is errors, which is like, um, a self-explanatory.

So you need to track the number of errors that's being logged against the system. So the personage of failed request or percentage operations, for instance, if you see a 4 0 4 L when you're an accessing website, so that needs to be logged down. So you, you need to understand what are the different kind of errors that being blocked when a user is accessing application.

And the last one is saturation, which means, um, how much capacity is being used So you have a fixed set of resource and how much of that is being used. Um, so resource when you say like CP or memory or, or whatever resources that is set up for the application. So you need to consider what's at what percentage is being used currently.

So then you can decide if you have to, um, um, upscale it to meet the demands or if you need to, um, um, if, if it's sufficient or so that sort of regular analysis. Very important to understand. So these are the four four, uh, golden signals framework that is suggested by Google now that we saw what is observability and what are all the, uh, you know, key golden signals?

And we also looked at the three pillars of observability. Let's take a deep dive into understand like what are all the tools that is needed to effectively, you know, monitor a cloud native application. So before, you know, even actually looking the tools required to monitor a cloud native application, let's take a deep dive into understand why we really need a cloud native observability stack and how it makes difference monitoring a traditional application when it comes to monitoring and observability, and, uh, how it is different for a cloud-based application.

So monitoring and application in on-prem data center is different from monitoring cloud-based application because it's quite complex to use the same traditional tools. And, uh, tools for monitoring have undergone significant, you know, Adam shift over past few years, and the need of observability arises due to unique challenges of distributed architecture. So like, when does specific tools often, you know, struggle to pace up with this rapid speed?

And that is one of the, you know, key reasons why, you know, industry or companies are choosing open source tools, uh, when compared to vendor solutions, not just that, that are, you know, pretty much more advantages when it comes to open source. Uh, one of the major key advantage is cost effectiveness. So it's highly cost effective and it helps you to solve the vendor lock-in situation.

So you're not, you know, locked into any kind of a vendor. Uh, unlike you're using a appropriate tool, uh, uh, think about a scenario where, you know, uh, you're using the tool within your organization for around five to 10 years, and, uh, you wanted to, you know, transition to a different tool, and the whole application would've been, you know, instrumented using the APAs of a vendor tool, or you might be using a vendor specific agent, uh, which will be, you know, pulling up all your, uh, metrics or, which will be exporting all your metrics to the, uh, backend. And it, it, it involves so much amount of, you know, changes in your application and the whole migration to a different monitoring tool.

It's a painful job. So vendor lock situation is one of the key advantage which the open source tools solve. And that is, uh, that is something that we really need to think about it, uh, whenever we are, you know, trying to get into a tech stack and, uh, faster innovation.

It's quite, you know, faster to innovate things. When you have things open source, uh, you, you can easily pitch in, understand what is the code and how it is, you know, being implemented across different companies as it is open source and transparent, and it is customizable. So since it is open source, there are a lot of things like, uh, you know, you'll have multiple, uh, blogs or you'll have multiple success stories and case studies, uh, or, you know, white papers, uh, that has been submitted across, uh, for a particular open source tool, which gives you a lot of confidence in order to adopt the tool and also in order to, you know, understand how the functionality of the, you know, particular tool is, you know, working within the system.

And last, but not the least, it is community driven. When I say community driven, uh, there is against, you know, quite amount of challenges where if you, you know, select a project which is not graduated, uh, when I say, which is not graduated, um, CNCF, uh, uh, is one of the, you know, majorly used open source, uh, foundation. Uh, there are a couple of other, um, you know, open source foundation, but, uh, most of the cloud native and, uh, you know, predominant tools like Kubernetes, Prometheus, all this come under CNCO Foundation.

So they have a couple of, uh, you know, stages when they onboard a project. So they start with a stage called Sandbox, uh, which is the initial stage where, uh, you know, a project has given an opportunity to, uh, you know, prove themself, try new things, address the solution, uh, for a given problem slowly, uh, once, you know, they make a, uh, you know, remarkable, uh, you know, achievement there in the sandbox layer. It's then the, uh, CNCF community, uh, try to promote that particular project from sandbox to, uh, incubation.

And from incubation, it moves towards the, uh, graduation. The name itself says graduation is the, uh, highest point for any project within CNCF and Kubernetes as one of the, uh, key project, which most of the organizations are using, which is a graduated pro, you know, project. And, uh, eagar is another, uh, you know, tool that we are going to discuss about with, again, a graduated project.

So it, it's necessary for a company, uh, to, you know, understand these stages because, uh, you have to pick the tools based on the adoption rate and, uh, based on how much community support you are getting in terms of that particular project. So that's, that's where I was trying to emphasize more about community driven. So when I say community driven, keep all these points in your mind whenever you are trying to build your, uh, you know, tool stack for your cloud native observability.

So let's discuss some of the widely adopted, you know, open source projects around this observability area. So starting with flea D, flu D is, it's kind of, you know, uh, helps you to integrate or unify all of your logging into one place. It helps you to collect and process all your logs, and it, it basically, it helps you to forward your logs in one place, followed by agar.

Uh, agar is predominantly, you know, famous across the distributed, uh, tracing. When I say distributed tracing, uh, the, the system is so, you know, uh, what, what I would say the system is so much complex nowadays. Uh, unlike a monolithic, it's a distributed architecture where have, you know, or 50 or sometimes like more than 50 microservices talking to each other.

And it makes, you know, highly challenging for an SRE or for a production support engineer, uh, to debug a particular issue, uh, without any end-to-end tracing. And that's the main area where distributed tracing is solving it, is trying to help you to visualize your end-to-end observability using, uh, concepts like tracing. When I, when I say tracing, it involves, uh, multiple other concepts like how to export the trace, what is span, uh, you know, all those things that we discussed about latency.

All those things can be graphically visualized with distributed tracing. So Agar serves this purpose, and if you're looking to build a, you know, robust distributed tracing system, uh, agar is one of the, uh, solution that I would suggest followed by Prometheus. Prometheus is a, you know, go-to monitoring toolkit if you are trying to do a metric based monitoring.

Uh, we, we just saw that, uh, how important is metric and using metric, there are a lot of things that you can be, you know, doing, like, you can be creating a dashboard or you can be creating a chart. You can create a alert based of the metric. Similarly, if you, if you keep on, you know, uh, expanding your, uh, functionalities around this metric, then, uh, Prometheus is one of the, uh, key tool if you are looking for metric based monitoring.

Last but not least is the open telemetry. I would say Open Telemetry is one of the widely adopted, uh, project, uh, like other open source tools that I mentioned. But this is more of a framework that helps you to standardize your, uh, applications instrumentation.

When I say instrumentation, uh, I think most of the, most of the companies get into vendor lock by using agents. So this is the area where Open Telemetry is trying to solve. So it, it tries to eliminate that agent.

And instead of you using agents, you'll be using open tele telemetry APIs and SDKs, and using, which you'll be exporting the telemetry, uh, to the, uh, backend. When I say exporting telemetry to the backend, it might be all those, you know, things that we saw, like it can be metrics logs, or it can be traces, but how we are exporting, we are not using any vendor specific tools here or agents. We are using a standard which is open to everyone, and that's make, you know, life simple when you migrate your tools to a different tool, uh, in future.

So, moving on to next slide, let's look at some of the case studies. Uh, when I say case studies, uh, we only look at some of good companies, uh, you know, where they are having a huge number of, uh, you know, applications or their ecosystem is quite big when compared to, you know, uh, basically, uh, fan companies. So we are going to look at, um, some of the case studies, how, you know, they're trying to proactively, uh, look the outages and it comes to metrics starting with eBay.

Uh, so transitioning to metrics first culture, but open source tools like Prometheus and Grafana, it helped them to, you know, ready 30 percentage of MTTR within 12 months of time. And Google being one of the pioneer of, uh, SRE principles. So SRE practices at Google led to increased automation, reducing manual toy, and allowing engineers to focus more on impactful and interesting problems, rather than getting, you know, looped into repeating or, you know, mundane works.

This, this is one of the, you know, uh, good, uh, you know, area that we can focus on, like SE is a big, big word. So when I say SRE, a lot of things within SRE, like S-L-O-S-L-A-S-L-A error budget, these are the key areas where that you need to focus on if you are looking at customer satisfaction, because, uh, SLA is a very key metric that you need to, you know, look at and you, you need to make sure as a company we are not breaching SLA and, uh, a proper alerting mechanism set will help you to, you know, achieve that kind of, uh, a good customer's, um, you know, uh, feedback or customer, uh, experience. And, uh, Netflix.

Netflix is using, uh, trust engineering combined with, uh, observability to monitor their globally distributed architecture. So this helped them to isolate their failures during the outages. So this is like from a high level, uh, but there are definitely, uh, a lot of other case studies and, uh, like all the companies are trying to achieve that, you know, uh, that is the end goal.

They want a proper customer, uh, feedback or customer experience. And, um, I, you know, having, reading this kind of case studies will give you some kind of idea to implement, uh, these kind of, uh, things within your organization. Let's look at some of the facts and stats.

So starting with efficiency gains teams with high observability maturity results, incident 50% faster than those without it. This is some, you know, uh, this is from Dynatrace and the, this is something that we'll have to look at as well. And cloud adoption impact 90% of the organizations using multi-cloud setups, seed observability as a critical to managing complexity.

This is from CNCF and coming to reliability studies sh shows that outages cost enterprises around $9,000 per minute. Observative releases, developers, time spent diagnosing production issues by up to 40%. And this is from Google.

Yeah, let's look at the recent trends in observability. So coming to the open source dominance, as I clearly mentioned about, uh, the different open source tools, which, you know, we discussed about like Prometheus and ger, uh, we also looked at a couple of other things, open telemetry as well. So if you, if you see the GitHub project and if you see the number of, you know, people commenting or number of people talking about it raising issues, you'll understand the dominance that it has taken across other projects and other tools.

So that is one of the key thing. And, uh, the study shows that open TER adoption has been increased over 80% in 2024 and coming with a per observability, uh, as all of you know, everyone is moving towards, or everyone is trying to include a and ML along with their existing tech stack. And this is also true in terms of observability.

So one of the examples, a and Bel Models is now helping to detect anomaly in real time to prevent incidents before they, you know, kind of escalate. And also predictive insights, enabling better capacity planning. So this is one of the, uh, you know, area that, uh, you know, people are trying to go deeper and understand how AI and ML can go hand in hand with the observability tool stack.

Let's look some of the forecast, uh, when I say forecast the future of observability. So the growth projections here that as per Gartner observability tooling market is expected to grow up to $10 billion by 2027. And, um, that's huge.

And, uh, coming to evolving role of ai, that's what we saw. Just, uh, the previous slide of like how a and ML is, you know, going hand in hand with observability. Some of the, you know, major advantages, like you can identify issues that might otherwise go unnoticed by a software engineer or by a production engineer using AI and automations because it's quite difficult that, um, you know, you keep monitoring this.

Definitely there is one of the other thing that is that's going to be missed. So using ai, definitely it's that you're going to be training the, uh, you know, model on your data sets or on the, I know, public data sets. You want to make sure that the model is able to perform the task, uh, and trying to help you, you know, efficiently manage your system.

It's not like you're going to give the control to AI completely. It's like you're taking the advantage of AI along with the, uh, you know, stack that you have built. So that's where, you know, uh, the actual, uh, advantage is going to come in and you can prevent problems before they arise, as I said, like by integrating AI in your cloud native logging and monitoring tools and, uh, the standardization as I mentioned, uh, we will have to, you know, always look at, uh, uh, in a solution which is standard and, uh, portable.

When I say portable, uh, open telemetry is one such example. So you instrument your code with Open Telemetry and you are free to be, you know, portable across any kind of, uh, monitoring, uh, tech stack. So Open Telemetry is becoming the universal standard for telemetry data collection.

Yeah. Uh, this is our LinkedIn handles, uh, both we know and myself. So feel free to, you know, connect with us.

Be happy to, you know, have a chat, uh, or if you want to understand more about observability or if you wanted to, you know, share something that you found it useful and we are happy to, you know, learn as well. So feel free to connect with us. And, uh, that's it for today.

Um, we are going to end this session. Thanks a lot for everyone for joining this wonderful session, and it's a pleasure to, you know, be here and, you know, helping you guys to understand more about absorbability and, you know, monitoring. Thank you.