David Owczarek – Navigating the Service Metric Swamp

This is texturing TV. It's actually interesting doing a conference when you're between jobs because everybody wants to know who you work for. I worked for Adobe for 14 years while I was there.

I did a devops transition. I did an SRE transition and I ran a lot of services the last seven years. I ran the SRE and a few other teams for adobe's document Cloud which included the online portions of the whole PDF ecosystem.

So if you open to PDF, I probably would have known about it and also the Adobe signed digital signature platform, which is an enormous Global infrastructure for obviously really highly transactional work and that has informed a lot of my opinions and experiences around metrics. And so today's talk is a bit of a journey because over the last couple of years. A lot of new information has sort of come out just practically speaking about how to do metrics.

So I'd like to wander through that territory with you. Please feel free to jump in if you have questions. I do have a fair amount of material prepared, but We had a small room here.

There's no reason not have a discussion if we have that opportunity. For those of you who are familiar with SRE Concepts. There's a couple of things that I want to Anchor Us in right from the start.

The first is this concept of slos slis and sla's collectively. I call these slxes as a sort of a general concept but the idea is that you have Telemetry that comes out that are indicators. These indicators have thresholds which are performance objectives.

And then you also have agreements you invade with customers which are service level agreements usually found in contracts and the world that I want to create has these all nicely elegantly paired in a way that makes sense across them so that you have one set of these slxes that allows you to wrap your application in coverage not only for handling availability events or other things that affect the customer experience, but also all the reporting that you have to do to manage the system and deal with folks. Now in terms of the actual signals that we're going to look at Google's SRE Canon also advises us there. They have the four golden signals.

So I'm going to steal liberally from this stuff because I find it really helpful. The idea here is if you're gonna monitor something these are the four areas that are the most important. However, I'm gonna focus on errors and latency.

And the reason is that those two are actually if set up properly direct measures of the customer experience and one of the things I learned at Adobe was everything has to be looked at in the eyes of the customer experience particularly. We learned this in the signature business where if you didn't Of your signature platform available people didn't get work done. They didn't sign contracts.

They didn't book Revenue. It's not anything really more Mission critical than that. Transactions are super important, but they're sort of an indication of volume not really necessarily the customer experience.

And while saturation is a resource ratio, usually talking about consumption can be predictive again really focused on the The customer experience but I'm going to flip the words around and make them positive instead a negative. So I want to talk about correctness and speed versus errors and latency and note here that when I talk about correctness. I'm not in the tier of like data science.

We're not talking about durability or Integrity of data or anything like that. We're talking about the fact that when you sit down to do something at a website or or to throw a command at an API, you expect to get some kind of response back that is correct to your perspective the way the system behaves and you want to be fast and that's the way I anchor the discussion of any of the metrics. So we're not going to get into the weeds.

There's more metrics than you can shake a stick at some of them will become important to become slis. Some of them will not that doesn't mean that we don't consume them or use them or look at them for other things. But I want to keep this real simple and simple and elegant as I can.

So I mentioned before we have slos slis and slxs and and we want to build, you know, an elegant version of how we can use these to manage the the services that we're running. So here are some requirements for that. I'm gonna pause for a moment because this is a very wordy slide.

Basically, I want this to fit into the our vision of the way like a good modern devop shop works. We have a cicd pipeline. We have automation all across we're able to use that to put data into sources that we can then extract for other purposes and there's very little hand touching here.

This can all be done declaratively. and I sort of Drew a market texture slide to represent this. Um, you know, this is a market texture slide in the sense that it just tries to show something conceptual.

I'm not actually worried about all the stuff on the on the right in Gray what back ends. There are what third party Integrations. There are actually we're discussing last night with some folks here how that is probably the biggest risk of risk that you have to availability is through some of these third parties.

But we're not really concerned with that because when you again go to a website, that's the endpoint you're dealing with. You're not worried about whether that transaction requires b or X or any of the other services, you're just record. You're just what answer am I getting back from the thing that I'm actually interacting with and to that extent with our our view of those things kind of ends at the monitoring layer.

We have Telemetry coming out that we're going to consume and that's where we're interested. Not in all the internal details of what's actually running over there. So you can see here the way those requirements are laid out and I will spend a minute just talking through how I see this working an example might be you know, somebody builds a new microservice.

They can declaratively configure slos and slis for that in the code that they're submitting in a config file the ci/cd pipeline knows how to handle that. It knows how to deploy that through environments. It knows how to update the monitoring system.

It knows how to change the alerting mechanisms to reflect threshold changes. Those rules can be used to drive a bunch of things. There's typically three places where I've found the most demand for information about service metrics at a very high level one is the internal demand to know how things are going and that's usually a dashboard in this case.

I'm calling it The SLX dashboard, but there's many many dashboards there people have grafana dashboard Splunk dashboards Etc. All this stuff used to understand the state of the system. I'm really trying to focus on the ones that are most critical which means the most critical customer workflows.

Those would be the things that would be good candidates for an SLI. But at any rate typically have to dashboard this stuff so that you can manage things internally. You also have an external obligation typically to tell customers how you're doing.

So this could be a real-time and my up or down and then historical. Well, here's what my availability was over time this kind of a table Stakes now for the SAS business and build within that is this idea of there is an SLA of record you have probably some set of numbers that you're committed to to customers and you need to have unofficial source of those so that you can evaluate when there's a problem if somebody comes after you for a remedy for example, do I have to pay or not? How does that all work?

So that data should come out of this same stream and if we can integrate all these things together, we get a really nice nice elegant view of the world. It's it's not particularly simple though. The second thing or maybe the last thing I would point out here is this idea that again in an ideal view of this these slx's could do all of the incident response alerting required.

And anchoring that concept is the idea that you are testing a customer function, you're testing the customer experience if that customer experience be crosses the threshold of acceptability as defined by your SLO. You are having some kind of availability event you need to react and so you should fire off your incident response mechanism to to make that happen. There may also be cases where you know, slo's don't have to have a single threshold.

So you could have a warning threshold that doesn't fire off the huge alert that just actually brings your on-call team into solve things again in a well-constructed set of metrics. I'm hoping that we can accomplish all of these with the same small set of things. And again, we're focused on two which are speed and correctness.

okay, so that's a lot of background kind of level work. We haven't actually even really talked about a metric. So let's start with availability because I've mentioned availability a few times.

What's availability? How do we get it from this stream of data? So let's look at how you calculate availability and we're gonna drill in from the top here.

There's three different ways that you can calculate availability. These all rely on this concept of having sometimes you're up and sometimes you're not and if you think about website that has say two outages in a month, there are segments where you're up and there's segments where you're impaired and you know up time over up time down time. It's just a sum of those segments.

The mtbf and mttr calculation is even more interesting. So I'm actually going to spend a little bit of time on that. Theoretically if you're calculating them correctly, it produces the same number.

But that may actually not be the case because I think a lot of people don't. Actually calculate it correctly and actually I would challenge the utility of almost all of these availability is required usually for contractual reasons, but it's a shell electric. It doesn't really tell you what happened.

It just tells you what the end result was in terms of the penalty that you're paying so, you know by say, well I had an hours worth of downtime and a quarter. Was it all at once or was it you know 61 minute intervals. It's you don't really know from the shallow metric that you get from availability.

So it's very tempting to look at something like meantime between failure and meantime to to restore or repair. However, you want to say it and say well those seem like really important numbers. So let's take a look at meantime to restore.

So what do I mean by meantime to restore? I mean from the moment that your customer is impacted on in an unacceptable way to the moment that it's fixed. Now the fault might have started before that there might be stuff that happens after that.

I'm just looking at the area where the customer was impacted and saying that's really the time to restore that from the time they were impacted to the time. This is service became nominal. That's the time the time to restore and if I if I sum up all of the outage durations in a particular time period and divide it by the number of outages I get this nice average.

The problem is that one thing we've learned in the last few years is that outage duration data is not in a normal distribution. And so this has been studied now in a couple of ways first at Google they had a very large pool of incident data. And Ransom Monte Carlo simulations on that data to see if changes to process that impacted mttr would show up in the measurement of mttr and determined that good percentage of the time it would not.

And the reason is because you have a skewed distribution. So your average in your median are not the same and also because you have a very long a lot of orders of magnitude involved, right you can have something that lasts a couple of seconds. You can have something to last a couple of weeks.

And so the idea that I shave a minute off of a process that should affect my MTT, but if my next outage is two hours long and I actually gonna see that the answer is most of the time you're probably not and so mttr and actually mtdf which I'll get to in a moment. I think a really better for their conceptual role in the conversation rather than their numerical role. There are ways to maybe to maybe mathematically correct for these things.

But ultimately you have to answer what what are the insights that you're trying to get. So mtbf is not helpful either and it's essentially the same formula for mtbf. The problem is that we want to report on accounting periods.

And so what my experience was I want to know what the mtbf was for last month or for Less quarter. So, okay. Well how many outages do we have and let me do the math and divide it and I'll tell you how many it was and this leads to a calculation that is not this summation.

It leads to a calculation that looks like this. Just say okay had a month. I had two incidents therefore by sort of think about that.

It's probably 10 minutes is my meantime between failure. So I have a division by zero problem, right if I have no instance in a month, it's Infinity. So I gotta add a one to it.

And what you're doing here is not calculating mtdf you're calculating really like that. The number of outages you had in the month. Yeah, and even if you were calculating mtbf correctly, which would be Averaging the sum of all of the uptime segments and respecting time periods.

One of the problems here is that outages rarely cross a financial time period right? It's rare to have an outage that starts at the end of the quarter on the in the evening and then lasts until the next morning but uptime period span those and when you chop them to create a calculation you ruin the value of the number. Okay.

So that was a lot of math. Here's the other way. That's just a little more gut for both of these.

You're trying to make a develop an Insight through these numbers into a system that is incredibly complex that you are changing constantly and it's behavior isn't going to be the same from one measurement to the other to the next either. What I instead like to do with these numbers is to say okay meantime between failure is talking about how stable how often how many outages I'm having if I try to optimize that I'm in a world where I can't ever have an outage and that to me represents thinking that it's kind of a few decades ago. The place that were in now is no I can have outages I need to fail quickly and and move on and clean up.

And so, you know meantime between failure as a way to I think sends the wrong message. It says we should build an enormous vertical system that's hardened to the point where it never scales. That's actually not the best way to do a lot of things these days.

So let me sort of summarize this with some words of advice from John olsbaugh talking about shallow metrics actually obscure more than they reveal and I think many of us have experienced this in our day-to-day life. If we have a difficult month we go to our leaders whether an engineering or SRE or whatever and we say what's going on and they can tell you well, we have the database problem and you know count ID problem and the blah blah, so people understand really where things are at. Okay.

So how do you actually calculate availability then? We just sort of threw away the ones that we think are the most obvious or the most common methods. So let's play Let's play an exercise.

If you don't mind to indulge me. I'd like to get your opinion on something. There's no right answer no wrong answer.

Here's the scenario. You have an application doesn't matter what it is website API. It has a 60 Minute event where it has a problem exactly 60 Minutes starts exact starts stops.

No, no slopes here during that time exactly 50% of the transactions fail and exactly 50% Succeed and they're all fast. So the question is how many minutes of downtime do you take for that? So I would actually like to hear if you have thoughts about that.

I will show you my thoughts in the next slide. I'm going to ask differentiating Lay It outages unplanned. Yeah, this is unplanned downtime.

I'm planned on schedule downtime. All right, I submit that there's Two answers that are sort of obvious from the beginning right from the start. One is obviously an aerator 50% is outside of Tolerance.

I've clearly got to have some kind of objective that it's lower than that. Most people would have an error rate threshold in the one 2% range and I was out of compliance with that threshold for 60 full minutes. So I should take 60 Minutes of downtime for that because I was not operating within the self-defined tolerances of the site.

Now this gentleman in the back who brought up like what's the business angle of things earlier? And this is where you sort of rubber hits the road because there's no businessman in the world who would agree with this they'd say, are you kidding me half of the people involved had no problem at all. Do you think that I'm gonna publish that they were down and they're all going to come running at us for remedies?

It doesn't make any sense at all. And again, there's no right or wrong answer here. You can take a purist approach and do the thing on the left.

You can take a more pragmatic approach and do the thing on the right it may vary for your business. I can tell you. At the in the signature business we looked at the agreement creation rate is what we call it.

Can I create a contract and can I sign a contract? I can do those two things. Then I'm functioning and you know, there are a lot of other things that that platform could do.

It had managed Pages, you could track transactions you could resend things. You could change signature methods you could do all kinds of things. But if you were creating agreements we were down and so while there are many ways to to to have indicators, sometimes you have to reduce an incredibly complex thing down to something incredibly simple and it was very unsatisfying but more times than not it was the right answer because it wasn't fair to say nothing.

Everything was broken when we had plenty of people who were doing lots of lots of important work. But it's not that simple. So I won't give you time to read this, but I'll talk over it.

Let's say that the 50% of transactions that failed were not evenly distributed amongst customers. Let's say that 10 customers actually drove all of the 50% errors and you had a hundred thousand customers who didn't have any problem at all. So now what's your SOA calculation?

So this gets very very messy in the weeds. And one of the most important Lessons Learned is that even when it's not optimal you really need to stay with the same thing. And so even though in our case using agreement creation rate was not always we thought the most accurate way to describe the impact.

It was the most consistent way from outage outage and therefore it was it was it was an easy way for us to talk and have a common language. wheel around is something that Something that works, so I'm going to focus on not using averages. for reasons previously stated but by using percentages or percentiles with one caveat, I didn't really have time to bake it into this presentation, but If you're doing latency, there's some suggestion that percentiles maybe not the best thing because it's hard to view them in aggregate.

Or math reasons and so those are best heat map. There's some great articles about about that out there. I'm going to focus on correctness because it's real easy to see this is what you would you know, maybe get from the error code and you can imagine a really simple scenario here.

Like I have let's let's leave the signature platform kind of complexity aside and say I have a microservice that's doing one thing. It's converting a to B. That's all it does and there's a stream of telemetry coming out saying I had a return code and I had a timing and really that's the data you need to do all of this right that gives you your correctness and that gives you your latency and so I want to look at that.

I have now introduced the idea that we're gonna check within a Time range. And of course this gets into how you monitor. We're not going to go deep into that except to say that it may be more effective.

Sometimes to have two ranges to have a short range and a long range. So Google advice doing one for five minutes and then another one for every 60 Minutes on Rolling basis so that you know inconsistent behavior in the smaller window doesn't show up but does show up in the larger window. So there's some ways to just optimize around things like that.

So these are very simple and it leads to let's let's look at a graph of what that might look like. So here's a hypothetical performance chart for the success rate of this. 9% SLO which means that if the if the success rate drops below that we are in and outage situation.

And you can see here that we have sort of two different events happening. And when I talked earlier about, you know, do I charge the whole thing or do I do I just charge who is impacted? You can see that very clearly here in the red area under the dotted line.

and so the sort of rule of thumb that comes out of this is if you can do that well Then you absolutely should and that gets you a ultimately a higher number. If you can't do that. Well, or you need to be extremely conservative you just take the whole time.

So this little notch on the left that was about 25% drop to say 75% success. So it's pretty easy to say, oh two minutes times 25% air. It's like 30 seconds of impact time.

The other one's a little harder because you have to figure, you know, can I actually calculate the area under that curve? Is that easy to do or not? And what what metric am I using to do that?

I'm sorry. We're using the available the error rate to do that. But is that data coming through in a way that it's easy for me to make that calculation?

Now the if you're in the Google world, you would say that the error rate is actually the availability metric. I like to take a more expansive. Look I just had talked about correctness and I should have put speed.

I wrote latency correctness and speed I think both of those are critical and so when I look at overall availability, I would actually have one chart here for correctness. I have another one for latency. And then the availability number is actually the sum of the downtime in the two.

So I'm taking a hit either way. It's really slow. I take a hit if it's not functioning I take a hit and I have to protect against both and the discipline that this imparts on the whole organization is well, we obviously can't take down time that impacts customers.

So we've got a really focus on this and since we are establishing what these boundaries are. It's in our interest to minimize this this red area. To the curve and that sort of goes back to what I was talking about before when I was saying well empty mtbf is about preventing outages how effective is preventing outages as a strategy doesn't seem that that much there's usually sort of a Pareto rule there you get you get a lot of benefits quickly and then it becomes very expensive and then you look at failing fast in other circumstances, but reducing the amount of time in an outage is also a way to increase your uptime and this shows it right here.

I need to make that area in red smaller so that I have a higher availability. So some learnings from this that'll pass on before we go to questions. The most important one is to think about the underlying data and whether or not what you're measuring actually makes sense for the data that you have.

There's a I've got some references on the last page here. There was an a really interesting report done by a company called ferrica who is building a service called void the varica open incident database. I believe it's called and what they're doing is collecting incident reports from companies who are willing to share them for the purpose of creating a public source of data to allow people to analyze what incidents look like in the real world.

And so they've put some of their data in they've taking data from AWS and other folks that publish them and they've had some companies they're working with directly that are willing to do this. And so now with several thousand incidents in this database they're able to actually do experiments on the underlying data, which is where some of this information about distributions came from so avoid averages and I guess the most important concept is that you're measuring a customer experience? And if you're doing that, right?

You can learn on that you can report on that and you can get a great deal of most of your sort of primary level coverage around metrics and availability right through that one simple construct. I've got references for most of the nitty-gritty details of stuff that I talked about here and I'm assuming these slides will be available afterwards. So be able to take them there as well.

Thank you.