Seeing the Observability Forest Through the Data Trees: From Data to Outcomes | DataOps Day

Hey folks, my name is Pramesh. I work at last nine io as an ss r e and evangelist. Today I'll be talking on seeing the observability forest through the data trees from data to outcomes.

Observability is the ability to measure the internal states of a system by examining its output. A system is considered observable if the current state can be estimated by only using the information from the output. Uh, it tells us what is wrong and more importantly, why it's wrong.

Observability is crucial in the cloud native environment of today as only an observable system is easier to maintain. For DevOps and teams, teams in for the state of the system plan for accreditation performance management based on the level of observability, uh, that they have with their software systems, it is even considered as a fundamental property of the itself as it can help us in making sure that the outcomes that we expect from the system are tracked, measured, and reported about. But when we hear about observability, and if we Google, uh, the term observability today, uh, you'll find tons and tons of blog posts and about logs, metrics, events, and traces.

Instead of searching for observability and getting results about observability, we always run into these, uh, data that is elementary data that the observable systems emit. We always run into that so much so, uh, that, uh, the first introduction to observability, uh, for a lot of people happens via these posts about metrics, logs, events and traces. And melt is just a popular acronym, uh, for, for, uh, all of these data types, uh, that observable systems.

M telemetry. The observability journey, uh, for a lot of organizations, for a lot of teams starts with instrumentation where data is collected from, uh, different microservices, APIs, uh, infor infrastructure components in the form of telemetry. Uh, it is used to understand the system, health falls, troubleshooting, degradation, um, exploring the health of the system, wire dashboards, and then consuming it via alerting with the help of different, uh, observability and monitoring tools, both open source, proprietary and so on.

Now, the challenge is that while metrics logs, trace are, uh, called pillars of observability for a reason, the question is do they really matter a lot in terms of defining the outcomes that we get from observability? For example, uh, if I start a new service and I don't have metrics and logs, uh, metrics and traces, I only have logs. Uh, does this mean that my system is 33% observable, or if I start using metrics tomorrow, does that make my system 66% observative?

Where should I go when an incident happens? Should I use logs, metrics, traces, or events? Like what should I go first, right?

I want to use metrics, but what's the cost of adopting it in my entire organization across all my services? What's the storage cost? What's the maintenance cost?

And most importantly, does it make my system more observable? Does it give me the answers that I'm looking for? I try to answer some of these questions today, and while doing that, I'll shift the focus from the data to outcomes.

I'll talk about monitoring versus debugging in detail, and we'll also talk about why they're different and how even the data that we collect from all of these software systems can also help us in specific monitoring scenarios versus in debugging scenarios. Why this matters today is basically because the workloads have changed. Uh, if you think about a typical Kubernetes three node cluster, uh, with specific 10 names versus fire deployments and a standard monitoring library like let's say cube set metrics, it emits up to 17 K metrics.

By default, 17 K times is by default. That's a huge number. Infra is mostly ephemeral thesis, like it comes up, comes down where it controlled by auto-scaling strategies and so on.

At the same time, services are iron. So even if you consider the typical, uh, Kubernetes workload that we saw, it has metrics from pods, deployment services, namespace, everything. Do we need all of those?

Is the real question. Can we get most of the information that we need, uh, from our telemetry from specific metrics? Instead of going through all of these metrics is the question that I will try to answer today.

It also matters because of the volume of the data, telemetry data that has increased. At the same time, the velocity, like the way it is generated, uh, the frequency at which it is generated has also multifold. All of this results into basically variety of data that gets submitted from different parts of our software systems resulting into complexity that we as a humans have to manage each and every day.

Essentially, all of this results into the cost, and this cost is not just the monetary cost, uh, but I would like to break it down from the perspective of observability where see is the cardinality or the number of unique data samples that we get across our software systems. Always the operations part where I need to have my team members or myself managing all of this, uh, telemetry pipeline. And the observability stack scale is where basically I want to make sure that as my system scales, as my product scales, my observability stack also scales, my telemetry also scales, and all of this resulting into essentially toil where I have to make sure that things are kept running all the time as my product is running, uh, uh, as well on the other side.

So all of this results into the, uh, cost of observability and telemetry. Now I have different choices, right? Should I create a separate metric store?

Should I use distributed tracing? Should I dump everything into log and just get away with it, right? What to do?

Should I use managed services? Should I use a on-prem service? Should I use a self-managed, uh, Prometheus or something like that?

All of these questions result into a decision fatigue that am already fatigued with, uh, while doing my other tasks. Now, I have to also care about what to do, uh, in terms of having these metrics logs, events, spaces, right? So that is what causes, uh, problems with lot of teams when they want to start their observability journey.

Now, in your thinking from the data perspective or the data coverage perspective, let's try to think from an outcome perspective. What outcomes do we want from our observability systems? Uh, if you think about different use cases, uh, at least for DevOps and SRE teams or even developers, they want to know the system behavior.

They want to know the application performance. They want to know when something is degraded, when something is under attack from outside, uh, outside like bugs or either via, uh, users trying to crash your system. Uh, we want to know to debug a failure.

We want to know that how a particular workflow, uh, is being managed inside the application flow. We want to find something which like find something bad before even it happens. All of these questions, uh, all of these outcomes also mean that we want to ask certain questions to our systems.

And those three questions fundamentally are basically like, what is wrong? Did we change anything because of which it became wrong? And what can we do so that this doesn't triple?

Basically, these are the broad categories of, uh, questions that we can ask our systems. And if we get answers to these questions, then uh, we can of course, uh, answer the, uh, or try to get the outcomes that we discussed earlier. Whenever an incident or a failure is detected, a natural inclination is basically to find the root cause, debug it right away, put everything together in a war room and find out why exactly this particular incident happened.

Although doing it in real time as the incident is underway can be counterproductive to the quality of the user experience that our users expect debugging its best as a post factor activity when the incident is mitigated. So whenever an incident happens, right, the first inclination should be to know what exactly has happened, who all are impacted so that we can communicate it to them and then try to recover from the incident, try to mitigate from the incident. Once that is done, then we can of course analyze it with granular information like law traces, we can debug it, we can find a root cause and fix it properly.

If you think about this from a high level perspective, the monitoring side is all about understanding the system health, being able to quick decision, decision making where we can quickly recover or mitigate from the instrument. And all of this becomes very time sensitive because we want to make sure that the system is back to itself, uh, very quickly. And at the same time, the debugging is all about root cause analysis, testing it so that it doesn't happen again, correctness, so that we make sure that the issue is fixed properly and it'll not happen again.

So that is all about debugging. Now, with all of this information about monitoring and debugging from the outcome perspective, let's look at melt again from a fresh point of view and see if we can understand which data can be used for which outcome. Let's start with logs.

Uh, logs are basically the easiest to start with, so I am also starting with them. Uh, they're basically a dump of how the system or programming is behaving in real time. Uh, they can be as descriptive as we want.

They can literally be anything like, I can just dump a low print world and that can also act as a unstructured logline. But there are also standard programs that emit logs in a structured format, and those are called as basically structured logs. So they are, uh, these can be structured logs can be emitted from different software components, not just our software programs, um, but also can be emitted from different hardware servers and so on as well.

Different languages, frameworks, tools, they have their own logging formats. This makes logs harder to standardize across services and teams. If you think about logs, volume, getting out of control, uh, because of a third party service change, its response time.

It's not quite unheard of. Like we keep running into those situations all the time. So when it's easy to get started with logs and the adoption is quite easy across teams also, volumes and standardization are two challenges that basically make log slightly crippled, uh, on how they can be used.

And because of the granular data structure itself, understanding the system health and system insights becomes very hard because the challenge is that data is so granular, you have to really figure out what you are trying to search for. If, if you think about many logging tools, right, like paper, trails, Splunk, they basically give you an ability to aggregate logs on top of the raw data, ask questions to that aggregated data itself to find out patterns, anomalies, and that is how you can, uh, essentially get the system help from log. But just getting it from the raw data is quite hard.

The another problem with logs is also they don't have really relationship information like how the service flow is happening across the system because there is no common, uh, pattern or common key that is present across all logs in our uh, system. You can of course add it, but it means that an added, uh, work for your teams, different teams, uh, to manage it for different services and infrastructure components. Then comes metrics.

Metrics are basically quantifiable measurement of system. If you think something like what metrics your, uh, apple Watch reports, that is the best analogy that I can explain. Metrics with Apple Watch tells us that my heart rate for last one hour was this.

I walked X number of steps, uh, yesterday. I walked y number of steps the day before. This is the difference between both of those numbers.

This is how metrics are represented. They're the fastest and cheapest way to understand the system because of the aggregated nature of the data. They give a bird perform view of the systems performance of examples of metrics in the software world are basically C P U percentage, easy to instance, Redis, database cache, hit rate, uh, or latency of an API endpoint.

The key difference between log and metrics, that metrics are aggregated already. That's why they can help us in understanding system health in the cheapest way because the volume of the data is automatically used. Essentially.

They also have dimensions, dimensions, something, but key value pairs, right? Key value payers tells us that we can slice and dice based on these attributes and their values. Did you encounter the recent S B V crisis, uh, that basically caused panic in the SA industry a few months ago?

Well, the issue was resolved, but how did the US authorities come about it? Using some key indicators that they were observing and that started deteriorating over time. They fixed the issue in few days, but continue to root cause it, analyze it and making sure that the real problem behind that issue was fixed over next few weeks.

The first priority was always to mitigate, right? That is something that we discussed earlier. The first priority for an incident was to mitigate it using the key metric indicators and then do a root cause analysis, uh, to fix it properly over time.

Metrics are essentially values that are aggregated over time with dimensions attached to each data point. Uh, they're scalers, so they don't have any dimensional dimensionality or relationship information, uh, attached with them, but it also makes it easy for standardization because there are different libraries, uh, that can be used to start emitting these metrics. And they can also help us in understanding health and system insights, detect patterns, find anomalies because of the aggregated nature of the data.

At the same time, debugging with metrics is not possible to the same degree that we can get it from logs and traces because the information is already aggregated. The relationships are also not presented metrics because the information is confined to a particular component that is emitting the metrics. Then comes traces.

Traces is basically a complete journey of a request or workflow as it moves from one part of the system to another. Uh, it is achieved by basically adding a common standard trace id. Uh, as the request flows through all the hops of the workflow, each time this context is forwarded to the next execution hub, a shared trace ID is basically passed around, which helps connect the dots, uh, at the end.

And this it process itself with called as distributed tracing. As we can see, like trace data is basically made up of different spans, uh, which has the context of each and every hop along with the information, uh, about the previous hop. And this also makes it very easy for tracers to have externality.

We can know that a particular request or a particular flow is going through different hops of the system and then it is failing at some point of time during this, uh, transactional flow and we can, we can figure out where exactly, uh, it is failing. So traces essentially provide the relationships and directionality between two data points. Now, if you think about our comparison point of view, getting started with traces is tricky because all my service, uh, owners, all my teams have to make sure that they're emitting the traces in the same format so that I can see the real benefits.

If you think about the previous slide, let's say the job queue is not having the trace id, then even if I have traces in my front end in my API gateway, even in my, that doesn't tell me the component picture, right? Because one hop during this workflow doesn't have, uh, is not emitting the traces. So you have to basically make sure that tracing is implemented across all services, across all components, and that makes it getting started and adoption slightly tricky.

Debugging is best done because again, similar to logic provides us the granular information and also relationships are present, but we run into challenges of volume increasing beyond a certain, uh, as well as not being able to really understand the health and system inside because of the same challenges of granular data that we saw. Standardization is quite easy with races because of standard libraries like open telemetry SDKs that can help developers, uh, like skip adding, uh, distributed tests, tracing in their code base by just adding like these standard libraries as drop-in replacements. And these libraries have auto instrument which can, uh, start the tracing immediately.

Tracing can also be considered as structured logging because basically the trace ID can be considered as a request id that is uniform across all requests, and you can also use it, uh, similar to how you can use logs by passing the request ID as well. Then comes events. Events is the most confusing one because some logs can also be event, some metrics can also be event.

So there are different definitions of events, uh, that different people have. They're hard to define basically, but I'll give it a try for today. So event is primarily a change event that something has changed, a port restart, a deployment, a configuration flag change.

Uh, events are important because they affect the system state externally and can help correlate incidents with other telemetry data that we are made, like metrics, log spaces and so on. There are two kinds of events that I'm interested in from an observability, uh, point of view. Basically one event is, uh, they happen over time like a performance frequency, uh, presence or absence of something, let's say average ops from the San Francisco airport, uh, in the last week.

And then the second is the event and its data of interest like when actually it happened, when was the last time arsenal within e p, that is an event that somebody might be interested. At the same time they can, there can be schema based events, domain events, uh, but essential point is they can help us in unlock correlation, uh, with respect to metrics, traces, and logs. Events can also have dimensionality similar to metrics.

I can emit key value pairs along with the event in the schema and I can have the same benefits that a metric has. So while getting started with events can be tricky because of the lack of standardization, uh, which also hampers the adoption and standardization, uh, parts, but it can help us in debug. It can also help us in understanding the system help by doing correlation.

Uh, the volume of events can be under control depending on which events we're capturing because sometimes those events are really high precision events and we have to make sure that we capture them properly. Uh, but volume can be under the control. So with all of this information, if we think about the answers that we wanted, the part that we started with, while metrics, events, log faces form the backbone of today's observability and monitoring tools and are the key pillars of observability from the answers that we want, there are specifically two buckets.

The monitoring bucket is especially the real time bucket where we want to know what happened. We want to communicate the relevant stakeholders and we want to recover from the incident. The post factor analysis happens for debugging, which is analyzing it, debugging it, using logs and traces, making sure that the root cause is corrected properly.

If you think about the personas who have to deal with this, uh, data and cases, SREs and DevOps are essentially concerned about monitoring where they're the front line of people, uh, who are in front of uh, the screens to understand if something is under degradation. Uh, communicate with the stakeholders, pull in the service owners, and then make sure that the incident is recovered while programmers, developers will do the analysis, will debug it root causes and make sure that it doesn't happen again. Another essential point that I would love to talk about is about the volume of data that we're generating.

We talked earlier about too much data being generated and that causing a decision fatigue in our, uh, decision making about which data we'll look at. 80% of elementary data is mostly news. We just emit it, pay for it, but not really use it.

The key point here is that we have to pay for this. There is a cost in terms of either the decision fatigue or monetary costs that is involved in maintaining that much amount of storage. It also results into slow codes, concurrent taxes, having issues underlying storage, not being optimized enough to address our needs when we really need them.

And it also affects things like real time alerting because, uh, if your data storage is not optimized, it's not, uh, fundamentally solid enough, uh, to answer the questions. All the real time use cases are crippled automatically. So for high precision monitoring and observability, you really need control over data and that control can only be given by providing enough control levels.

The idea behind this is that you already know the shape of data. You already know that certain metrics are useful for alerting certain telemetry data is useful for ss l o Certain data is only useful for long-term compliance storage, but if we give you enough control levers on top of that to manage the data, to manage the storage, to manage the query, to manage how the data moves around, that will help you in terms of using the data effectively. So if you can treat the alerting workloads differently versus if you can treat the exploration workload differently, not just from the storage perspective but also from the query perspective, query guarantees perspective, that can unlock a lot of choices that you can use the data effectively.

Things like having automatic data tier where certain data is moved into a different fast storage automatically versus a long-term data is moved into cold storage where it can only be accessed at a granular, not at a granular level, versus having policies that I want to access the fast bla here while alerting policy, which means that only the alerting workload can access that data versus the exploratory data can be accessed via another policy. This gives you enough control over how you wanna manage the data. All of this essentially leads into declarative observability where you declare how you want to access the data using tiers, using policies, and that makes sure that your full control over the outcomes that you want from the data.

So instead of data being the primary citizen, you care more about outcomes. And with these control levers, you can define those outcomes in a declarative way so that you can get enough value from all the data that is getting emitted from all the laws, metrics, cases, uh, from different components. That's all I have today.

Hope that you like this, uh, presentation. Uh, if you have any comments or questions, uh, I would love to take those and would love to discuss more about, uh, monitoring, observability, debugging and how we can use, uh, most use of our metrics logs. That's all.

Thank you.