DevOps, SRE and Observability – Three Sides of the Same Coin – The SRE Show EP 5

It is. Hi and welcome, everyone. Appreciate you joining us for another episode of the SRE show.

We have a great topic. We are talking about observability and SRE three sides of the same coin. We'll play on devops and SRE and and observability here with my co-host partner Austin Parker.

How you doing? Austin? Good.

How you doing? Good good. You are your purpley dressed in all black all black background.

You got the vibe man. Just posting from my cave. From the cave.

Excellent. Well, we cave we've got a an amazing panel. I can't wait to get started here.

And here's some perspectives on the kind of interplay between observability and the SRE role and why don't we do this how about just have folks do a brief introduction of each of yourselves. And then awesome if you want to kind of kick off the conversation, we'll get things rolling. So, how about Michael?

Would you want to start us out first? Sure. Thanks Mitch.

Hi everyone. My name is Michael Keough. I'm a senior staff security engineer at confluent before that.

I was a senior staff SRE at LinkedIn working on everything from the traffic stack to low-level databases. And of course all the observability in between. Fantastic, and how about Andy?

Yeah, my name is Andy grabner. I am a deaf rail. I would call myself like that at Diamond Trace in doing this for the last 14 years and trying to help people understand.

You know, how observability can actually help the weather the devops necessary and thanks for the opportunity today to share my thoughts. awesome and Jessica Hi, I'm Jessica Care on Twitter. I'm just a Tron.

And I really care about systems thinking and how that applies to development and how development is contributing to it in particular. I'm excited about learning from our software. Which is why I work in observability.

So I work at honeycomb now. Fantastic. Yeah, good day.

How are you? Hi there good. I'm gonna stop developer Advocate light stuff prior to doing this.

I was a chaos engineering expert at turned us SRI turn chaos engineering expert and I come from a self-taught background constantly thinking about how to make our systems more reliable and continue learning from them. Excellent. I just want to thank again the team at lightstep Austin for sponsoring the show.

It's a pleasure to work with you all of you. Once you do a proper introduction of yourself too. Oh, I'm Austin Parker.

I'm head of devrel at lightstep. I Can I have an interesting way into the industry but I'm focusing observability because I think it's the most powerful tool. We have to keep people from getting woken up at 3am and trying to figure out what's going wrong with our systems.

And I think there's a lot of reasons why you can get woken up at 3 am trying to figure out what's going on with your system. Not necessarily all Technical and base are not all based in technology. And so great Community calls like that for sure.

I know what you mean and most and most of them actually aren't necessarily due to technology. That's the interesting part. Mmm.

Very true. Once you kick off our topic kind of this interplay between observability and the yes, there you roll if you will kick things off. Well, just to kind of I guess to start.

It's interesting. I think when you look at you know. the history of sort of words in the sphere, right what we call certain things observability in SRE have kind of had Covenant growth in a lot of ways we people can point to the Google, you know site reliability engineering handbook that was published as being one of the drivers, I think of organizations adopting observability practices And I also think that it.

The flip side of that is true, right that observability helps Drive SRE practices in the organization. So one thing that actually want to speak to something that just brought up is that Almost in a way. These are kind of drifting apart right where observability is starting to become more something that is focused that like the developer level versus being strictly in the vein of sres.

And so I think a good way to talk about the start off is ask the question like is observability and monitoring something that is really just for sres, or is it something that has a pretty broad based application across the entire software development, you know the sdlc as it were. I think this is one place where a devops has a lot to teach Dev. Because observability is essential for SRE when your troubleshooting something you need to be able to see into it and you need to generate the knowledge.

You need at the moment and developers like have this idea from college days that they should already know everything. They need to know about the language and the software and we're Continually recognizing that that can't be the case. We need to generate that knowledge.

And this is where observability can expand from SRE and help us. I think from my perspective. It's it's interesting how observability evolved because I know a lot of us have worked in the space for a long long time.

I remember when I started in the obstability days was 14 years ago. We still called it monitoring everything started with APM application performance monitoring and it was a very tough thing to actually get observability into a system right there were just a couple of vendors that build very specific tools agents that got visibility and I think it was also back then I guess not cheap to get that visibility and I think the great thing about where we are now 2022. We have a lot of great open source saying that's a lot of great open source tools.

It's easier accessible and therefore we also get more visibility not just in production. But as you said early on in the development cycle, you know integrated in your development tools and I remember and this is something that kind of higher for my bucket list in my life. Because when I started analyzing distributed traces the first time 14 years ago, when I started my journey in that area, the number one problem that I saw why systems were either slow were failing or not scaling were basic things like inefficient access to vacant systems.

The N plus one query problem was always something and making data inefficient database calls in the loop. And I've showed so many developers, you know how easy it is to fix these things, but they were kind of surprised or really my hibernate is doing this. I thought hibernate is the magic thing that will make everything efficient when I existed data and then really it's making a thousand calls to the database.

I didn't know now. However, I think we have the tools in place baked into existing runtimes and Frameworks that highlight these things right away into the development idea. If I think organizations are leveraging these tools right and educating the developers like when they get started and so kind of coming back to my Could at least I hope I never in the not so distant future have to look at the distributed Trace where I see the M plus one query problem because it never made it interproduction system crashing your website for instance.

Absolutely. I think we've all seen that Meme of the burning house and a little girl saying it worked on my desktopper. It's fine now and as an industry, we've sort of frowned away from that sort of thinking between developers and sres to where production is a shared responsibility.

And to Andy's point over the last 10 or so years. We've built the tools that allow us to have you know, Common metrics being emitted from our systems. So, you know developers don't need to sort of understand the custom implementation of some sort of metric is standard across the board and so as sres, you know, we can help teach them those basics of what those metrics mean and how they sort of work and really up level our partners to ensure that before it goes into production.

They've got the understanding and the means to go and understand what the application is doing before. It is fully deployed to production I think to to Michael's Point too is like as a series we have trained our teams like hey, you need to set up like best practices with those ability. This is a stock for organization, but I think at least in my experience the last five years, we've also gotten better about teaching them what it is.

It's like let me actually give you a little bit more of an overview of what we're looking at and why these are the metrics and the systems the parts of the system that we're actually going to care about and that allows for them to actually carry a little bit more of what they call like skin the game because it is their code running on production and that they actually start carrying a little bit more of it and that they want to help get better in a sense. was interesting when things that you all made me think about is In the development World, we're constantly abstracting things away right with languages with services with you name it. I mean it's that's part of what development is about doing more with less code in some ways, but that also abstracts the way the details kind of what you were.

I think you were talking about Andy of well, you know, you're making tens of thousands of calls the database. Did you realize when you like did that one thing in I my experience has been the best developers. I enjoyed working with the most Were the developers that had a pretty good knowledge of kind of the full stack and I don't mean that the marketing term full stack, but you know all of the software that they're working in from the OS to the database to the network to infrastructure software.

Not that they're expert and all of it but to kind of design Great software you really have to understand a good bit of that. Not that they know all of it and all detail but seems like that's part of the Gap that we're talking about is how do we fill in that? You might have a distracted that way in your code, but here's what's going on.

And here's what we need to do to solve a problem or make it perform better, whatever. It might be true agree. Hibernate makes your life easier as the developer and you need to know when that's a problem and observability can give you a clue with that.

I think it's funny though. I'm gonna actually argue against myself here for a moment because I've it to the point that There's my early right about Open Standards. Right?

So we have open Telemetry now and the observability space to provide a standard framework for creating generating and then exporting, you know, representing Telemetry data like metrics distributed traces logs. and the objective of open Telemetry in my mind is to not have a external agent that you plug in in order to you know, get that Telemetry data, but to be built into the underlying runtime or Frameworks, you know for hibernate itself to emit distributed choices and metrics right that you as developer don't have to think about. How do I get this Telemetry?

You just have it kind of for free Gratis, right? But that is also an abstraction right like we keep you know, the more we kind of push these details and you lied these details from the developer. Yes, we're giving more information, but we're not necessarily teaching them.

Hey, this is how you actually represent. Your system right? We're moving away in some ways.

I think from having this kind of idea of craft where you as a developer responsible for going in and writing the code that defines how the system should operate and what we expect the system to do and we're we want to push all that off into these again these agents these invisible instrumentation packages. I don't know if this is like a bet. You know, I don't think maybe this isn't a value judgment, but I do think it's an interesting, you know destination or what seems to be the destination of a lot of what we're trying to do right now in observability or even an SRE teams, right like the concept of like oh platform engineering and you're just providing a platforms to your devs and they don't have to think about any of this stuff.

They just they use the platform. Everything's great. Is it great like is that actually upskilling Developers?

Well, I think open Telemetry but only source one problem open Telemetry solves the problem of how do we get the data? And I think what I think with you are alluding to is is if you are a platform team or if you are providing a platform to your organization, you need to figure out what to do with this data. And I think this is also about a value prop still comes in from from vendors right like a couple of vendors here on that on that panel.

You know, how can we make sense out of data and not have to maybe teach developers in school how to what every single metric means but the platform itself that interests the open Telemetry data then alerts on strange Behavior right coming back to hibernate. I would expect that if hibernate is properly instrumented and the hibernate Community can say hey and here are some patterns that I would like every observability platform right just to automatically detect and say this might be something to look into because making a hundred or thousand database calls from a single distributed Trace is typically not what we see. And so I think this is but for me open telemetric as I said, you know softy how do we get to the data?

Because this was always a challenge where at least we got Challenge from our customers they say hey, why do I need to install an agent? I don't trust you. Well now the power is back to you.

You can instrument based on open source libraries. So you solve the data capturing problem, but let us still solve the data analytics problem or you know, whoever the US is candidate open source Community as well. But I think this is this is what this is what the next step is open Telemetry only solves one problem of the whole thing.

Yeah, absolutely. I think you know we do have that sort of data problem solved but it is sort of how do we build the layers of abstraction here. So going back to a database example earlier if we have high 95th percentile latency, we need the tools where we can then break that down for both sres and developers to understand.

So is that coming from a select query and update query and then, you know go to the next level to actually go and look at those queries and those run times and by the types of queries. Those are the sort of observability tools that we need in the community so that you know, you can have your layers of abstraction for Simplicity, but you can also sort of go through each one to actually understand what the underlying problem is in an efficient manner. the beauty of hibernate implementing it's open Telemetry instrumentation or as hibernate specific instrumentation of it is that that those spans can express what's important in that library and it gives a library author the opportunity to tell the operator as sorry your developer.

Hey, look this is what's going on. There's a story there and then Andy makes the point that like interpreting that story saying is this normal is something we don't have a generic solution for To do and I think to Austin's Point earlier were the it was mentioned that it's like we're just building platforms and we're building a lot of Attraction layers. Like I do agree with you the amount of abstraction layers that we build sometimes is very questionable and like it makes it harder to debug.

We're not up leveling our Engineers like you said, but at the same time like I think of what it was like to be developing software at Uber where there was already a thousand 200 microservices that's like having standardized platforms of like this is the way that we're going to be using reaction scaffold to build internal applications. And then these are the ways that you're gonna containerize it and send it and make sure to use M3 for for metrics and ability like because there was a guided Well, it got created a set of Guides of how to actually go from. This is my idea of a application.

I want to build internally to get this all the way to production to integrating with the rest of the marketplace if such things weren't created. I think the developer would not be able to make an impact within six months to a year. So to an extent like in bigger Enterprises or larger startups like things like this do need to happen more organically in order to upload those Engineers to actually get their job done.

And are you are you arguing for? The structure kind of brings more productivity in your balancing sort of the artistic nature of how you want to design software versus, you know, you can't build that whole stack yourself it already exists. We're all sharing a similar approach on architecture.

Whatever it might be. I think the way I see it is like as you take the Walk of building your organization, you have to make different trade-offs every single month and you kind of have to think of like what type of Engineers do I want to be hiring for do I want to be hiring folks that are going to scale out of platform to then? Allow folks to just come in a little easier and just get building or do I want a lot more building builders out there that are going to do things like for one-off one-off cases versus try to make a larger impact.

I think there's like no right or wrong answer in my book. I think a lot of it is just the way that you that you want to lead engineering teams. Yeah, you're trying to increase productivity and and you have to know what's there right?

You have to know what you can reuse or if there's a certain pattern or you know part of the stack that you're using. I'm curious from a from an SRE perspective. is open till imagery sort of delivered on the promise of open Telemetry of standardizing that the data model and the access to that data and has that has that been a big Boon for us yet or is it still?

in its development and we're waiting to see some of the benefits from that where we in in our you know life of open Telemetry and what it's supposed to do for us. I feel like I just got started on my open Telemetry data, so I can't like sorry let open Telemetry Journey that I can't like speak for like how it's like a hundred percent. So folks wanna comment go ahead.

I mean the completely biased Source here. I actually I think it's it's funny because I have said this publicly several times. the like level of adoption of open Telemetry right now is completely by mind boggling when you think about it in the context of where open Telemetry is at as a project and where It can potentially be or where.

Where we wanted to be in five years just to pick a number, right? He's saying it's good, right the adoption is good. I mean I'm saying the adoption.

it's high but it's surprisingly high for how rough it is because I think this is the thing that gets Maybe lost in the sauce a little bit is that open Telemetry is designed almost as a frame as an observability framework to build observability tools with not necessarily as your like primary way you interact with it. So a good example is there's open Telemetry Java the Java API and SDK and then there's the open Telemetry Java instrumentation package and really when you get down to like how open Telemetry should work is more stuff like that where you have this this agent basically that you drop in and it handles a lot of the setup for you and instrumenting your hibernates and your Springs and this and that and the other and then you use that in conjunction with some helpers like the ability to annotate your code, you know a little at trace or whatever and get you know, extra spans and and use the API to kind of add some attributes and stuff to those but What we've seen instead is a lot of people actually just using like core open Telemetry and going out and like writing instrumentation code and it's like that's cool. That's fine.

You can do that, but I don't think that's how you should be using it. You know, we want this to be built in to the Frameworks and the software that you're using so examples where that's done, you know dot net integrates open Telemetry into the system Diagnostics fcl or BCL actually, sorry. rest and early both integrated into kind of runtime level tracing packages or Tokyo, I think in the case of rust which is you know, Very conventional way to do this for that language and it is a native experience and a lot of other packages use it and so you drop it in you hook up open Telemetry and now the traces and the instrumentation that you've built.

In their language with that sort of native low level access to what's going on can now be expressed admitted to a bunch of different places for analysis. So In five years. I hope there's a lot more of that.

You know, I don't necessarily I I actually don't think in five years people should be using the hotel SDK quite as much as they do today. It should really mostly be done through. Again different levels, you know integrating it into the language itself or to the language runtime or to your framework or some some abstraction over open the moving parts of open telemetry.

And I'm curious because there's a bunch of others are ability people on here. Feel free to disagree with me. I am merely a voice in the desert.

Now I agree with you because and I know meet you to the beginning we should not have a panel but we just agree with each other. But I agree with you in every in an aspect that the people that build these runtimes no paste right what two instrument and what is important and in the past we as vendors to build agents that did runtime instrumentation on the Fly. We always had to reverse engineer and head to figure out what is actually based even though we didn't know the random as good as the engineers of those runtime.

So I think that's definitely good thing where I think it's interesting though. You say it right? Let's use the Java or instrumentation agent.

So bringing it in another agent that is doing runtime changes. And I wonder how that will then go in the argument of what people now saying, hey, I don't want to install an agent dynamically modifies my code. I want to have it all under my control.

I think it's going to be interesting piece. Well so far it's going poorly because people complain when you're like, oh, yeah, you should use Hotel. and they're like, but we only one agent can be installed at a time and I mean eating growing pains, right?

No, my you know, my question is also why do I trust the hotel agent that is doing the instrumentation more than some vendors that have built agents for the last 10 to 15 years. People like open source. They like to be able to look at the code and feel like they own it.

Hmm. And because it quite crucially means that they can tell the vendor to go shove it when they don't like how much it costs. Yeah, right.

Yeah. No, I understand. I mean did you know open Telemetry for us?

Also his vendor is great because one of the the biggest the biggest challenge is for us is right now keeping and maintaining our our agents to do the instrumentation because there's a new runtime in a new library coming out every day. And so putting this to the runtime vendors into Library vendors and the framework Windows makes it also easier for us to consume the data and you as well, right? I mean we don't have to deal with this anyone this is great, but it's still it's still interesting from a discussion perspective.

No, I don't trust you because you build an agent that does something. Yes, they can configure it but I still don't trust you but I trust this piece that somebody also wrote that I don't know but I trusted now yeah, I understand the argument of Open Source and I can look at the source, but I'm still wondering how this discussion turns out. I mean, I don't think there's any reason that you should I mean some might argue that So it's interesting, right?

Obviously the obvious reason that hotel has as much of Interest support as it does because people accept the argument that instrumentation's agent instrumentation agents have become commodified and there's no actual difference between the data dog age and the dinotrace agent and you know honeycomb b line or whatever. Because it's all effectively the same code doing the same thing. It's trying to observe, you know, hook into the behavior of an existing library and on, you know on transactions start on transaction and you know, there's only so many things that a SQL client can do there's only so many things that an HTTP or grpc server can do there's only so many things that the Kafka client does.

Vendors I think correctly realize that the millions and millions and millions of dollars that they spend every year on their agents and on the care and feeding and maintenance and updates to them is a cost. They probably don't want to spend and it would be more fiscally responsible if they could convince the open source Community to do all that work for free. You know, that's the game for the vendor.

Right? And I say this as I say this well literally wearing a vendor hat. It's an interesting and this is actually an interesting learning we had from open tracing because I was in maintainer on that as well.

And one of the reasons open tracing was designed the way it was where it was just an for background open tracing was a vendor neutral API for writing distributed tracing code, but there was no implementation of it. So if you wanted to implement open tracing you had to create a implementation that supported the API. So there was the Yeager Tracer which supported the open tracing API and there was the lights that trace it supported the open Tracy API and data dog their APM Library support the open tracing API and it's really implementations of this API.

And the reason it was done that way is because there was this assumption by the designers of open tracing that vendors would not want to They would want to customize things. They would not want to just say oh well go use this open source thing help you but we don't own it. We don't control it.

Right. It's that level of like, I think this is in some ways. This is just the market that we're selling into or the market of people that are would do this in the first place.

They really don't care that much because you know things like Cooper we're so used to that now. All right, like as an industry, we have moved beyond the need in a lot of cases for Ironclad support guarantees, and you know proprietary code, you know, this isn't 2000 and we're not buying everything from Microsoft anymore. We accept that our software is running on things that Are out there publicly.

We realized that we cannot buy the illusion of perfect or perfectly supported code. Yeah. It's it's the catheter on the bizarre and the bizarre one, but I guess With tour guides, maybe we're in The Matrix, you know, we're batteries.

Yeah, this is super far off topic but this is a super there's someone that like grew up like someone that kind of got into technology like in the night you in the late 90s. Or the mid to late 90s when I was a teenager and like reading about open source, right and reading about, you know, reading this then in the Art of Motorcycle Maintenance and reading the cathedral limbs are yes. f*** yes are by the way but like these are transforming, you know, these are like transformative ways to think about things at the time and they don't really get discussed a lot anymore because we have this default Assumption of like, oh open source one, but it's really interesting to interrogate that and be like, well did it actually win or did we just replace the the cathedral with a different Cathedral?

There's a pot. There's a podcast for you. Yeah, this is another episode someone else somebody real quick question.

So so in in the world of SRE who owns open Telemetry and the observability tools when someone is implementing, you know, a collection of tools and some open source technology. What's the role of the SRE in that did they tend to own big parts of that? Are they helping instrument things?

Are they more of a user of those tools? What's what's the pattern that we see having developing or already in use? I I be interested.

In other people's opinions in this what I tend to see is I think this is changing there used to be the sres would actually wind up doing a lot of the actual instrumentation work or the ones that were kind of responsible for the instrumentation, but I think that is starting to shift and it's kind of getting shifted left a bit into the developers being responsible for instrumentation. But again, I'm Is what other people have seen in the panel? Yeah, I definitely think that there is that shift for developers to earn the implementation but there's still a responsibility for sres as the systems Engineers that people who sort of understand how all the puzzle pieces come together to sort of set the requirements of have something should be implemented or instrumented instrumented rather and then to also run some of those systems that you know collect that data and then display it.

Oh look on it Etc. There's still I think generally the expectation that we earn the how that data gets used. Throughout you know the engineering ecosystem.

But anybody else what's your experience? I think it heavily We rise depending on who you talk to. Right?

I mean, there's no one-size-fits all software engineering company. That's why I think it's hard to say who owns it. I still I also see like Michael just said the shift left not just because it's a cool marketing term that we've overused for the last years, but I still think that at least what I see most organizations is that sres at least are there to say hey in order to run reliable systems.

We have some experience and in order for you to do this. If I if you want your application to run in production, you want to make sure that you know, we get availability monitoring we get the basic monitoring of things that can go wrong. We want to make sure who's responsible so that we know if something fails we know who to You know contact so I think it's there is a Delta really enforce the kind of the best practices also strange term, but they could practices and them were S3s can build this into platforms the better the skills because otherwise you need a lot of sris.

for a lot of Engineers and then we can automate into tools the better it scales, but just my my observation but in general there's a lot of flavors in organizations with responsible for what also historically think who initially brought monitoring and Typically monitoring came in for some business critic Labs, then it kind of started to evolve and then this team at the negative experience and then this team became responsible for monitoring and observability. And sometimes they have managed to escape this and kind of handed it off, but sometimes not they're still the central monitoring team. that everybody goes to Thank you.

For more eloquently saying everything I was thinking. I've seen a lot of people working on implementing honeycomb and they're like, I'm the observability lead. Or I'm the observability person and on one hand observability is an emergent property of the systems and you can't centralize that in a person but somebody's got to operate the collector.

And also as bright information to the development teams on how to do instrumentation and how to look at the their traces and graphs and what to do with it. So maybe it's these monitoring teams that have now changed names to observability, but also I see a company's hiring for observability in particular So at least in large companies, it's coming its own, baby. I was gonna meet mention that I've usually seen Andy's way as well where it's like a nursery is the one bringing in like what is going from monitoring to observerability or just more of like these are the best practices and of course is they're able to like platform or ties like their best practices easier it is but I was recently helping a friend look for a new job opportunity from sis.

Admin to sree and I was going through job postings and I was surprised to see them on a webservability specific job postings that I was Finding, but at the same time I kind of see it as this is a very nice to have because like once you're comfortable and and like obserability it's a lot easier to go in and helping her organization get this done. Right? I'll like as like that's the specialization because you have knowledge of one what best practices are people you can contact but also it's a lot easier to get the job done due to that muscle memory.

So I think we're we're starting To see a good shift and then industry same thing with folks are working reliability. They they try to stay in that focused because it's like covering a lot of like strategy decisions that can happen in an organization. But it makes sense to the size of the organization.

Right and it's gonna dictate a lot of one person inviting multiple roles versus specializing. It's and tell me if I'm wrong. It seems like observability.

Certainly from a NASA represent. It's not a set and forget, right. I mean, yes, we need to update collectors and you know do or activities of kind of keeping things running.

But the applications are behaviors are changing what we're doing. We're rolling out more kubernetes. Now, we're doing it in multiple clouds instead of one or we're trying serverless in our app or whatever.

It is. The things that we're observing are changing. So it seems like while we can shift things left to kind of developers Implement that instrumentation.

There's also knowing what to look at and what how to understand it and and you know actualize that information do something with it in a change and refinement. I mean as you look at your as you look at your yeah as you look at your your traces and your aggregations and say I wow, I wish I knew this. Customer ID or whatever and then you go back and you add it.

There's also like a refinement process. Through actual you know. Human beings facing process changes, right like as you're doing, you know, as you're refining your organization your refining like how is information shared and how those are on call processes iterating over time.

And how are we handling retrospectives? And how are we handling all the the really important, you know, the important people to people part of actually doing work. Like that also.

Is something that you have needs to reflect on your observability practice? And how are we taking the things we're learning from observability and putting into that. How are we taking, you know changes in on call or changes in incident response and having that, you know having a process to go back in and add instrumentation or change instrumentation or so on and so forth.

I do think that there is like we touched on that with the whole who maintains The Collector thing. I think that really what we've well we have is that there is two there's actually two parts to observability. There's the part.

There's the system modeling part and then there's the secondary question. How do I like the pipeline right of the obserability pipeline cost efficiently, you know to store this cost effectively and query this and make decisions about what part of it goes where and how are you know, what's the fiddly bit parts of getting alerts done and so on and so on and so forth, so This is that I don't think this is a realization that is quite made it to sort of job racks yet. But I do think that there is kind of two part two big parts to it and it's that.

Instrumentation and pipeline are really separate questions of observability and the way you solve those and the teams that are responsible for those are very different. I would put that in the platform team. That's really yeah.

Yeah. I see observability teams often attached to platform teams. Mmm, I guess.

Yeah, but this just work for you. Yeah, I mean, I just see it as like the idea of an observability team to me is like the idea of a devops team, right? Right.

Yeah, exactly. It's an emergent property of the system. You cannot actually centralize it.

You can have a team that that's it more than enabling team that helps other teams. Yeah move toward the healthier system, but I think the the again the core part of instrumentation almost isn't a I guess there's also like two this is also a subdivided question, right because there is actually instrumentation and metrics and things that like an SRE team or like a more Ops focused persons. Don't care about that the developers don't True.

So within each of these spheres, you know within SRE within platform within sort of the more tooling and internal able inside you have people that want to do instrumentation for their tools, but also for their resources and then on the you know, the more pure pure Dev side it gets a little bit more into just doing instrumentation. Your custom instrumentation. We're going to customers relation.

But again like the modeling right? Like how do if I'm the developer if I'm if I'm a group if I'm a team and I'm sitting down like we're building our cool new service. That's gonna do X Y and Z.

How do we represent that? Like, what is this? What are the attributes that we need to put on there in our spans?

And in our metrics in our logs? How are we writing these logs to be? You know, how are we writing this Telemetry to be consumed by humans and machines.

How are we defining slos for you know what we're comfortable with in terms of the actual objective, you know our actual objectives but a better so on and so forth like that ideal state representation should not necessarily be enforced upon you like it shouldn't be an observability team coming in saying like, oh, we're gonna write all this implementation for you the sres the platform people should be saying here's kind of the roadmap or here's the guy like here's you must be this tall to ride right? You got to have this much time. Yeah.

Here's the minimums. Here's the shared. Here's the shared language.

We're going to use in terms of attributes and making sure that you're all world going in the same direction, but then when it comes to the actual implementation part that's really should be on individual Dev teams. And I think that advocacy for that really should be almost more of like a at a developer level right? Like you need Dev teams to tell her to have teams is like, oh, yeah, we we get through our backlog so much quicker, you know, or it's like when when something breaks and there's a ticket it's like it's so much faster for us to fix it because we went through and we did the work and we understand the system now and we have this Telemetry to show for it.

And that makes our lives better. That the state of devops report says teams with good observability practices spend more time coding. Very young there.

You can't I can't argue Adora. I'm not gonna argue with that for sure. Well, we're just a few minutes left for almost at the end of our time together.

I'm curious Austin. What are your thoughts on where where do we go next in terms of SRE and observability we talked about the instrumentation of application the developer might be doing maybe more of the platform part of that and then also sort of the the operational aspects of setting up observability and but I thought you also raced a really good point about it's still young and growing and maturing. Yes.

Everybody's not 20 years old, right? It's still early days for all of this and I think the you know, the next big thing is really just shifting. There's both shifting at the low on the high end right?

It's how do we make it so that team so the smaller teams can start off on the right foot and have a clear path to you know starting out with traces right versus logs learning these good but these best practices Building Systems with that in mind and then on the high end, it's how do we make it easier for large organizations and complex organizations to adopt best practices for their existing, you know, production apps because you can't just tell I mean In many cases, it's literally an impossibility to go back in and write instrumentation and you know in existing application like the code either doesn't exist or people have access to it, or the way the organization is designed is not conducive to it. So those that's kind of the two paths. I think the small scale like how do we start off the right way and the large scale?

How do we make this achievable? For orgs that are doing sort of a transformation. But I'd be interested in what people think too.

Any parting comments thoughts? I definitely agree. Go ahead Andy.

No, if you comment on him because I have another wish for the future. So I think there's a solve that the easier the journey will be because everything grows from there. One thing that we haven't really discussed today Beyond sort of collecting the data but is what we do with the data.

So, you know, we get traces we get alerts Etc. And there is I personally think it's still a gap where We don't have sort of very standardized run books or even order remediation run books of what we do with that data before we go and wake somebody up. So if you go and have you know, say a high spread pool utilization or you know, High P95 database latency what steps does your various observability alerting tools take before you go and Page someone at 2AM all of us here.

I'm sure done on coal dislike. It don't like being woken up. And so I think there is still a way for us to go as an industry to sort of Get the machines to do some work for us before we get woken up and have by the time we are awake, you know have all the information sort of all these pre-triage steps and all the data there for us.

So we don't have to go and spend an extra hour or two am in the morning to go and work out what's going wrong. For pictures worth a thousand words a nodding panel by everybody on the pan. It's worth about 10,000 words.

So Andy, do you want you to share one more kind of wish and then we'll wrap things up on that. I think this was kind of going into the similar direction, right? Because what do we do now?

We are flooding people with a lot of distribute the traces and they need to figure out what other parents we really need to look at. So what I would like to see is besides framework Windows instrumenting it with open Telemetry. Also providing some rules some patterns that then the observability platforms and consume the data can analyze and then say hey, Seems that your framework your runtime is not acting based on what the vendor of this framework around them wants it to be and then it can as you said either suggest a remediation action, which is highlighted to developer that they made a bad implementation and not using the library as intended.

Because as I said in the beginning, I don't want to when I retire at some point, I don't want to see this to do the choices anymore with the N plus one query problem. Should be something from the past. Second time you've mentioned that exactly that's why it's really important for me clearly.

It's well, thank you all all of you. It's been a pleasure great conversation, man. I learned a ton of things.

So I'm sure audience picked up a lot as well too. So Jessica, thank you Michael Andy, and of course and Austin great partnering with you on our SRE show was gonna say essary observability edition of the show episode. So thanks for joining us, and we look forward to everyone coming back for our next conversation.

We'll see you soon.