Reliable Applications – Emily Arnott, Blameless

This is texturing TV. Hey guys. Thanks for the throw.

We're here with Emily Arnett whose Community manager for blainless and we're going to be talking about reliability insights. And the first thing we're going to do is have Emily explain. What do we mean by a reliability in science?

Because you know that can mean a lot of things for a lot of different folks. So let's jump in Emily. Welcome the show Ah, thank you so much for having me.

So reliability insights. That's kind of a general term. We like to use for both a feature of our platform and then kind of the motivating ideas behind it, which is in a nutshell you should be able to easily understand the reliability of your system.

So Engineers, they like to get into the nitty-gritty they like to look into the details of each and every individual incident. But if you're working up the chain to your managers to the sea level to your investment boards, they just want something that's easy to parse is the system generally reliable. What kind of outliers have we been having our ideas working to make things better?

And that's what reliability insights gives you so our future is a collection of customizable dashboards and tiles that can draw from all sorts of monitoring data throughout your system or retrospective reports, you're giving your incidents and the overall goal is to just have one place. You can look and say yep things are looking good or oh maybe some changes need Do you think that we are more focused on reliability these days because we are a shifting of things like microservices and Cloud native applications. And the assumption is is those things are more reliable, but we know an absolutely know that for certain.

So we looking for more metrics and you know proof of reliability. I would say those things can certainly they can be more reliable, but then aren't necessarily more reliable right away as you start adding Cloud native architecture as you switch to kind of a microservices distributed model, the more third party Integrations, you have those are things that can help you but it also makes it much more complex to figure out where our problems coming from. How are things looking overall?

So I think you're totally right that the potential is there and we kind of need to rise to the occasion by using tooling and getting these insights to kind of compensate for that additional complication. Um, I would say reliability is also certainly more important than ever as we switch to kind of like a digital first economy where having a reliable robust digital platform isn't necessarily now a competitive differentiator. It's now just the floor.

It's just the expectation. Nobody's going to even consider you unless you have something that they can count on 99% of the time to work. Are there a set of best practices for tracking reliability or their metrics in particular that people should be looking for and what do they look like?

So you'll find it varies quite a bit from company to company but usually you can kind of start with the same building blocks. So something like mttx metrics. They don't really tell you the complete story.

But looking at you know, your meantime to detect an incident then your meantime to respond to that incident and then of course the meantime that it takes to actually Implement a working fix and figure out the root cause of the problem those are kind of the the foundation of the sort of things you're going to be looking at and then for each individual organization, you can start customizing that with parsing down the metrics to okay. What's the mean time to detect for this type of service? What's the mean time to detect when we're integrating this third party tool and that starts giving you a little more of a Nuance picture that can actually help you identify.

Oh, geez, all of our slowest laggiest incidents are coming from this part of the code base that creates a problem. So there's no one size fits all solution. But starting to just kind of look at these fundamental timelines across all of your incidents and then narrowing it down into kind of more problematic areas.

It's generally an approach that'll work for everyone how predictive can all this get. I mean, ultimately I think I'd like to get to the point where I'm gonna get a message that says, hey here are three things that are likely to get you fired and you do something about that before it happens. Well, we caution against it being too much of a you know, solitary guidepost that should always be contextualized by what was happening during those particular incidents.

So let's say you see some big outlier. And at first you're kind of panicking it's important to look into, you know, all the mitigating circumstances. Was there another incident happening at the same time was half your team away on some in-person Retreat or something.

There's always kind of a story behind the story. So it's not like we're saying look at one graph and then use that to decide the future of the company. It's more like this is what gets the discussion going that you can see these kind of outliers.

You can see a general Trend and then you can say okay that's where the digging into. Let's see why that's happening. So it's not so much that you can ever say oh, these are the three things that can get me fired right away, but it can start having that discussion and you can you can Certainly think okay.

I'm not going to miss something that that's that's a that big of a red flag. That's always going to be highlighted somewhere in the data and then we can take a look at it. We hear a lot about chaos engineering these days.

Should I just deliberately break something to figure out what my reliability is. I know that goes against the grain of a lot of folks. But you know is that when we need to do chaos engineering is certainly like a very Cutting Edge philosophy and it's certainly has benefits but it's not something you can just kind of dive into willy willy willy-nilly it's got to be really deliberate you have to have no what you're looking for.

You have to have your experiment area set up in such a way that it as closely as possible mirrors your actual production environment. You don't want to fall into the Trap of running a test that ends up not actually teaching you about how things would actually go you have to be really holistic. You have to incorporate, you know, not just the test and the vacuum but think about how would people actually be responding if it was real are people going to be unavailable.

Is there going to be conflicting priorities? So it's a real challenge getting that to be one to one between the Engineering experiment and reality and then it's it's tough to really make definitive conclusions. So our philosophy is kind of like you're probably already experiencing incidents, you know, nobody system is perfect.

You're you're probably actually generating quite a lot of data just from the things that are naturally going wrong. So we see that as a very important first step to start capturing all of that learning before you start needing to deliberately do chaos experiments and interject this artificial data on top of that. Do you think AI has a role to play in all this as we go forward?

We are collecting a lot of data. And of course the minute anybody collects data, somebody wants to build a model. So do you think that that's in our future or is that here now?

That's a very fascinating question. We are seeing something that you kind of see the umbrella term AI Ops and it's one of those terms that sounds very exciting and everybody rushes to call themselves AI Ops and then you drill into it and it's kind of difficult to pin down exactly what the AI is doing exactly how it's helping. You're right though in that anytime.

There's a lot of data being generated and you can start to look for Trends you want to kind of use the great statistical mind of the computer to get ahead of those Trends and I think there's certainly a lot of potential there. Um, one thing that's interesting is do you know about GitHub is doing that like coding assistant thing now sure where it'll help you kind of predict what code you might be typing by looking at all of the other GitHub repositories. So there's definitely very bold steps being taken in like computers can help us design systems and not just kind of execute systems.

So I'm certainly curious to see where that goes but like everything else I think in the short term it's going to be Of unreliability before it becomes a solution to unreliability that this is just another complexity. This is just another kind of black box that's maybe difficult to look into and that it really only reinforces the need to have a really solid foundation of reliability before you start experimenting with AI assistance on top of that. So we may discover one day that the code we're writing is not reliable before we put it in production, but we got a ways to go just yet.

Um What is the role of sres as we go forward? I mean are these folks really Engineers or they kind of just the newest flavor of an IT administrator that we're given ourselves a little self-promotion and race for and what do you think will be the role between you know, what an SRE does and when an IT administrator used to do and what's the balance? That's a really fascinating question.

And this is something that certainly every Sr. You will have a different take on everybody sees their role a little bit differently, but I think that's actually a strength of sres rather than an ambiguity that's gonna slow them down personally. I would like to see sres step further and further into the left of the cycle get earlier and earlier into development one of my colleagues Jake England made a really good comment.

Once that he said srees these days are kind of like mechanics they fix cars when they break down but he wants to see sres be like civil engineers. They're helping design the motorways. They're helping make things so that cars don't break down so often so I'd like to see more of a big picture role for srees.

So in terms of like it Administration that again is more reactive. It's more dealing with incidents as they arise coming up with policies for incident response since things like that, but sres I think have the opportunity to get involved in the actual design of the code and be thinking reliability first right from the initial design specs. I think that really kind of opens up our understanding of reliability that it's not just something that happens afterwards.

It's not just something that you're scrambling to put out fires, but something really proactive that doesn't catch fire in the first place, right? What do you think the state of observability is as it relates to reliability using your civil engineering metaphor or used to drive down the highway with a paper map and kind of hope to figure out which end of this was gonna work out correctly and was a 50/50 proposition today. I have a GPS and you know for the most part I get close to where I want to be and usually it's the last you know, two or three miles that are a mess.

But at least I'm in the neighborhood. What do you think is gonna happen with observability and reliability or these two things kind of join that the hip hmm. I really like your analogy here and if I can extend it even further, I think our risk these days is that we're building a lot of tunnels.

And maybe in the tunnel you feel like well, I just have to kind of go straight but you've lost signal you can't check your GPS anymore. And as our system gets more and more micro service based more and more third-party based. There's more of these dark tunnels that we're just kind of trusting.

All of our data is flowing through. So I think observability right now is a bigger challenge than ever and that it is linked at the hip to reliability. If you if you can't see what's happening in your service.

If you can't, you know take its pulse and check its Health, then you don't really have a sense of what's reliable. What is and where to make improvements. It's a challenge but the tooling and the best practices and people's commitment to it.

I think they're keeping Pace with that challenge. I think one thing that's gonna be interesting is Do you know the idea of kind of like white box observability white box monitoring where you're kind of just stealing with your system as if you were a user you're making requests and seeing how they go I think as systems become more complex that becomes more important because it becomes more linked to the actual user experience. Like let's say you can detect some sort of anomaly deep within your system.

That is a cause for concern but it's not as an immediate concern as a user just saying the services down for whatever reason I cannot use the search I cannot log in that's where your priority actually lies and making your users happy. So I think this idea of observability in the end you want to see through the eyes of your user you want to empathize with what they're trying to do and what obstacles they're having it kind of simplifies the process and it arrives at you know, the the problems they're gonna have the biggest business impact. All right, so not all my friends are as reliable as each other and shifting metaphors a little bit here.

Well, we have the same idea about applications and an understanding that you know, I still love my friends regardless of how reliable they are. I still love my applications regardless of how reliable they are. I just know what to do about the fact that one's less reliable than the other and we're gonna have to come to terms of them.

That's a very interesting perspective. because we always kind of think of If somebody has a bad experience, they're gonna look to a competitor. So the idea of someone being a little resilient to that personally and thinking okay, it's down today, but usually it's up but I think what's important is that when you have that sensation, it's because you already have a lot of trust right?

Like these people are already your friends sure. They might flake out occasionally, but there must have been something there to build up your trust that they're not just gonna flake out every time going forward and they're not going to flake out in an emergency. Right?

Like it's one thing for a friend to say, you know what I'm skipping brunch today. Oops. I drink too much last night.

Everybody said that experience but if you say I need a ride for the hospital and the same friend is going wow, so it's it's kind of about what expectations we have and what's critical to us in the moments. So I think you're right that we need to have a very holistic. Picture of how users expect reliability from their apps, but in the end it's a friend that flakes out all the time isn't going to be a friend for long.

Or at least you'll know what to expect and that's part of the issue right there. I think you know if I got rid of all my friends and applications that were 100% unreliable or even half unreliable. I might not have many left.

But today at least I know where they are and I know who to count on and with the call for and what situation Emily thanks for sharing your insights today and thanks for being on the show. Absolutely anytime. Thank you so much have a lovely day.

All right and back to you guys in the studio.