Establishing an SRE Practice from the Ground Up – The SRE Show EP 7

it Hello and welcome to the SRE show essary show is sponsored by lightstep service. Now company got some great panelists here. My co-host Austin Parker.

How you doing? Austin? I'm good.

Harry done. I'm doing well. It's always good to be with you and this fabulous crew.

We are going to be talking about how to establish or experiences from establishing and SRE practice function. Whatever you might want to call it in your organization kind of listen, Maybe Lessons Learned maybe experiences that we've had all the above. So before we get into our topic again, thank you to the good Folks at light step for sponsoring our show.

Let's let's do a little bit round of introductions and I'm gonna have Anna actually do the topic. So let's make her go last and then she can start the topic off Austin for folks that don't know you want to say just a little bit about yourself. Yeah.

I'm awesome Parker head of developer relations at light step and author speaker. born with all quite a few things no good at microcelebrity as we as we discussed on the before the recording button was in. Yeah, we're having that conversation great term.

All right, somebody jump in who wants to go next? I'll go. Hey, I'm Adriana.

Vallela. I'm a developer Advocate at lifestep part of that. I mentioned a platform team and an observability practices team at two cows have a black background in software engineering but devops and I found each other like eight years ago or so and it's been a love affair ever since so now I've since expanded into SRE and observability and it's been awesome.

All right, who wants to jump in there next? please I'm Jason Harley. I am currently a senior software developer at honeycomb prior to doing that.

I have done a variety of jobs in Tech but multiple tours in operations or site reliability engineering or devops and a leadership and an IC capacity. Decided to be here today fantastic great to have all. Irene yeah, I'm afraid I just maybe I'll see this.

I just finished the my term of two counts and I'm moving to a New Journey starting in one hour. my son So after this, I'll be signing in the laptop. They knew the email and stuff.

So that's what I call being in between jobs literally. Yeah. Well good.

Thanks for thanks for having the time to Fitness here between jobs that was pretty short stint. That's actually like a pretty Kick-Ass way to start your first day. It's like you walk into onboarding.

It's like what have you been up to? It's like, oh, I was just speaking somewhere like you hire someone that knows what they're doing. I'm bad.

Yeah. Thanks talk about making an impression. Right Honest if you could go and then let's you can kick off our topic.

My name's Anna Ana Margarita Medina. I'm a Staff developer Advocate like that part to joining light stuff. I did developer.

Advocacy a gremlin or help companies like do chaos engineering and that meant like help them in bark in the rest Journey sometimes and prior to that. I actually did S3 at Uber and like some Cloud infrastructure work, which is where I came about the world of us three and reliability because prior to that I was that engineer that it works on machine doesn't really matter. Really know about Ops and systems.

So it was always kind of nice to like get slapped in the ground and it's like now you need to understand systems and SRE and like be on call but like it became a lot of like learning and like trying to understand architecture and like why decisions I gotten made which was very much different from what we're here to talk about today. Like how do you go about actually like starting something as a small startup? I kind of got a chance to walk into an organization had already established sree, so there was some principles that were already in play and this is right before the sreem like SRI Bible had come out from Google but a lot of folks are spinning up this SRE team had come from Google.

So it was like a kind of interesting place to be in Very cool. So who would like to get us kick off the next step? It may be a little bit about your introduction to SRE.

How did you first get engaged in? I first got engaged by coming on to be an intern at Uber and little did. I know that that was actually their first intern in the start reliability engineering team, which meant that they didn't really know what to do with me except they expect me to know a lot and they expected me to be on call and things like it which was an interesting process because it didn't really it didn't really get thought about I did like onboarding week of engineering understanding systems how to like commit code like policies and then second week was like onboarding into chaos engineering at Surrey and infrastructure and in my third week, I was already on call for the chaos engineering service, which was like a tier one tier two critical service and then I was put on secondary for the infer rotation that manages our 40 infrast services and maintain like uber up.

So it was very much getting thrown into the wildfired and figure it out. So when the pages start coming in at two three in the morning for the like you destroy the chaos engineering service, I was like, I have a run book but nobody has died in me on how to run on call like whatsoever from like the process of like, how do I let people know that I'm getting page. So these are the actions that I'm being taken and like this is my first time that I'm getting close to touching Production Services for such a large scale system that as I'm executing stuff.

I remember freaking out I'm like one of my girlfriends in Miami was a wake at that time and she's just like can't you just copy the files that you're about to delete over so that you have a backup in cases needed and it's like that's not really how it works. Like if I do delete this like this just gone so it was like very much thrown into the trenches, but it's made me kind of like fall in love with the world and talk about like wanting to be better about onboarding people into these systems are really matter. Our customers and not just say like go figure it out on your own.

Cures anyone else awesome if you want to jump into kind of your you've seen probably more than one introduction of SRE and your career. I'm guessing Oh, and I when I got into SRE they called it Ops. And they call it this admin.

Yeah. I started out in it kind of in the You know originally like the 90s. in an extremely different world then we have today and when I got into software kind of officially in the 2000, you know late 2000s 2010s.

You know SRE wasn't. A thing like it is today certainly and probably not even the time that you know, Anna was hired. You know, we there was this new devops thing.

Right and I think our management tried to get everyone into it the same way that a lot of managers did which was to buy everyone a copy of the Phoenix project. And just kind of like put that in everyone's desk one day and it's like, oh we're doing this now. And as the person I I will say very similar story to Anna though where I kind of just wound up one day as like, well you've been doing the most work on this.

So now it's your responsibility to be on call for it and we didn't have a rotation and we didn't have really Any kind of concept of just like being off-call. It's like oh you own this. So it's your problem now, so if it breaks at 3:00 in the morning then If you don't fix it, then then someone's gonna ask you tomorrow.

Why wasn't it fixed? What that really led me to I think was that that's how I discovered the cloud and discovered like the actual utility of the cloud and I think that's shaped. A lot of wanted shape my interest in observability because one of the biggest challenges of that system was that it was extremely undiagnosed, you know, undebuggable there was very poor observability into it.

But second was the concept of you know, utility infrastructure and auto scaling and kubernetes and all that good stuff. So, I may be got there in a slightly different way and I I wound up going in a slightly different direction. I I will say the one thing that seems to be a commonality.

I feel like for a lot of people and I'd be curious what the rest of the panel, you know, at least we have two data points now where it's kind of like, all right. This is your job now have fun. No no life jacket.

Is that a very common way to be introduced to SRE? I'm kind of getting the feeling it might be but he also have a common experience you onboarding by trial by fire. yeah, I I also started out as this admin and you know, that was my first my first real job would have been in 2004 in like the infrastructure group of a large software company and SRE didn't exist outside of Google to my knowledge anyway at that period of time maybe it wasn't even talked about then.

I don't quite remember the timeline but After going through like all of that is trial by fire right like the thing blows up and that's the accounting system and that's you know, I remember being fascinated and Drawn to SRE as a concept and a practice though because it brought. Reliability out of that nebulous bucket of non-functional requirements. Everybody loves nfrs.

They're the best, you know performance bug free secure and now all of a sudden we could talk about reliability like it actually mattered to the business and that was just like a revelation to me. I remember reading the Phoenix project in groups about like the train wreck stories and and if you are here's your yeah your introduction to Um, yeah for you know, it's funny. You mentioned the Phoenix project and I remember the first time I read it.

I'm like, how do you know my life? It was like, you know, you kind of Revel in the in the train wreck. I I did like that book.

I know it's not turning into Phoenix project review. But the one thing I didn't like about the book is that everything was like in a little go at the end and everything's like la la all the problems we fixed but I feel like real life is a lot messier than that. the I've not had like a quote unquote like traditional SRE role but I've had like I guess SRE adjacent roles.

I started working with teams towards improving practices that would would help SRE teams. So but did a lot of work around just improving Employment Practices on kubernetes. So I did a lot of like research around Argo CD just to with the idea of like, you know, kubernetes clusters are so heinous to to manage that.

Yeah, you know, we need a nice overview like holistic vision of like what the happening in our kubernetes clusters learning. What's not running what's successful? What's not I'm not sure if we'll see when we get Adriana back Ibrahim.

That's that's the perfect opening for you. Yeah, so Okay, so we got what's in a big company where we had to devops Ops infrastructure all in 20. It was a really bad experience to have like for their living people anyway, so in that team we're responsible of services.

And the good thing in more management it was they changed and nutrients. So when they keep technology book came out, we kind of shifted to the platform style and and slowly we were giving services to support. And then from that came more SRE role of keeping up this production app up and everything is okay and then I transition to another company where it's totally sorry.

But I would agree with Austin it was like Hey, here we have this service. Issue here and nobody would know how there's no wrong books and nobody will save you and you'll have an email every maybe or pain after every half an hour. What's happening.

You don't know what's up. He just survived there. That it was a very problem introduction.

Yeah, yeah, so tip one for how to establish an SRE practice from the ground up don't do any of that. That's our first takeaway. I think look before you leap.

Maybe I do love an anti pattern. Yeah. Well tell you how to do this properly by telling you all the things you shouldn't do which is the best way to learn.

I think one thing that's interesting that even really touched on there is Because you use the word trend, right? And I I remember I was very distinct memory actually of being of going and visiting a client of my last company and they they were having a lot of implementation problems and they were wanting to learn from me like well, how do you all run your software right? and because we I was responsible for you know, this fairly massive, you know, fairly large deployment like 30 unique environments with a bunch of different configurations.

It was a platform as a service and so we had to run a lot of it as sort of integration environments. and I got we got shipped down to King of Prussia to meet with these folks and kind of share like, well, this is what we do in terms of how to actually operationalize the the how to run this And there was a lunch that we had and I can't remember. I don't remember where but it was an okay restaurant in a strip mall, but I remember what the topic of conversation so I was like have you heard about this SRE book?

And it was pretty new I think at that point and it got people talking about like well, this isn't what we're doing at. All right because there was and that's one thing that kept coming up in a lot of these conversations I would have and I think in a lot of people's even today like sort of their lived experience is that People that's already gets rebranded. Right our system administration keeps getting rebrands.

He keeps getting new coats of paint based on whatever the trends of the day are so you'll go from you know, Now the systems are devops people. What do they do? Well, they go and click on things and abs console.

And now those people are srees and they're going clicking on things and AWS console or their writing, you know, a couple scripts here and there and One thing that I think is maybe an important detail if you are thinking about starting an SRE practice is how do you actually Define SRE? Right? Like what is the portfolio that SRE has the distinguishes it from just a software engineer?

Because if you are a startup if you are a small company. Quite a few things that you would Norm that normally, you know, you might be called upon to do as an SRE aren't necessarily problems that you're gonna have right now. And if you bring in someone that is, you know used to this kind of trendy version of SRE where it's just another coat of paint on Administration, you know, they might not have the right skill set for you.

or they're going to try to introduce a lot of overhead that maybe isn't really super necessary when you're you know, a 10 person startup for example, so question for the group Is SRE something that you should try to hire for is this something you should have someone come in and build as like a a thing or is this something that you should kind of have your engineers figure out like well, what is actually what do we actually need? What are the things that? We care about in terms of site reliability and in terms of a practice around it and then either kind of having people shift for roles into that or or finding.

You know, maybe external candidates can come in and and map to that vision of SRE. I think a big part of that depends on the goals and the struggles with the business at that moment, right like for a 10-person startup. And we are interested in SRE and if we're at like this Crossroads of is this like a full-time role like we're gonna hire a specialist.

Or can we can we as an organization sort of start to adopt these practices? If I was in like an influencing position at that company, I think I would push for the group of 10 to start to think. About reliability as like a function like going back to you know, I said non-functional requirement.

That's pretty damn important. Like, you know, your your stuff working is what your customers want and so like and peeling that you know, what one more time what's even prompted this conversation? Like are your deploys bad?

Are you shipping features that are absolute blood baths when people go to use them. Like what is the what is the forcing function of this conversation? And I know I'm asking more questions, but I think but I think that I think that context really really matters because every company's God I hate the word Journey but I'm saying anyway, every company is Journey is is gonna be a little bit different because it the context of how you're running and what you built it with and what your customers are and what your business goals are are ultimately gonna influence.

All right, so but From what I have seen. 10 people starting to think about reliability as like a business function and you know how that can be measured. I think we'll do a small startup far more Justice and hiring a specialist.

I I would agree with you. The only caveat I I'd add is making sure that those people are willing to shift their mindset around that right because I've seen so many organizations where they're like, we're gonna become an SRE organization and then everyone's like okay business as usual and no one's actually willing to change right? So making sure that those those people that you're plunking into your SRE organization are gonna be the ones that are willing to make that change happen and I I would even say to a certain extent might be nice to have like someone Maybe One external hire for starters just to like inject some new ways of thinking into the team.

I think that's a good well just to Pivot on that question real quick. I mean is SRE something that should be approached as like a bottom up or a top down or I mean third options. It has to be both.

I think both because you got you got the doers right? You got your sres doing the things but like if you've got upper management, that's like let's start necessary organization and then You know and then they basically want you to keep acting the same way as before. Well, that's no good either right there there needs to be buy-in and it's got to be more than just words, right?

Because I've seen that so many places where they want to just pay lip service because it's trendy and some you know, they attended some like retreat where they're like, oh you should do this. So like we like really getting like getting those those upper management folks bought in is is key. You just call that the airline magazine article that the manager read way back home, right?

Okay, we need to do that's right. That's right. I feel like it ends up being both definitely because from the start Point like do you need those Engineers to be able to like level up like well not level up but like surface like the reliability issues that they're having and at the same time like they're going to need room to learn like it's not that they're just gonna wake up one day and be like, I know how to answer me.

So you either need like money to either like consult with other folks give their room for them to do experiments and like learning and pocs. But that comes from the Top If the top is not really bought in and as a function, we're gonna give you space for going to give you engineering time. We're going to be comfortable with failure because we are implementing new structures like those type of movements like are not going to be happening either.

So it does fall into like it depends but it kind of does come into if you have your upper leadership saying we're buying S3 like you just start throwing money at vendors or working this space but no one's really like oh these were the issues we were having and this is like how we go about using the tooling that we have and at the same time you can be uncovering all these issues and if you're upper leadership does not care or doesn't hear from the customer the pain points, they're not really going to take action on it, which I think actually ends up being like an interesting space to be in as I've worked that like SAS vendors and reliability space because you ended up realizing that the engineers don't really know how to even like Talk about the issues that they're seeing within their infrastructure. Like they can feel the burnout. They don't know how to recall they burn out.

They can be agitated at like having to constantly be fixing Genki test but not understand like where they need to commit the time or how to like write up a proposal of like we could be more proactive about incident by following these procedures. So sometimes like it was an interesting room of like coaching them on how to advocate for their needs I think on that point that's an interesting interesting question. Like is there actually a Sweet Spot really to kind of make that flip from You have a team that's following SRE practices or that's thinking in a reliability mindset and then transitioning to like well now we have a defined role and team foresight reliability.

Like is there a number of Engineers? Is there a number of customers is there I mean what are the obviously there's no hard and fast rule, right. Every business is different.

Every team is different. Every organization is different. But what are we what would we say the characteristics of what are the signals of like?

Okay, you have a reliability mindset and that's great. But now you need to think about resourcing and Staffing this appropriately. Gabrium.

Yeah, I really think in my opinion. The way software operates over the way people sell software it matters and how you do it. Sorry, for example subscription model companies are different than companies like contracts with big names on because that would take how you fix it.

And when there's a problem they just patch and don't talk to me just and then all of the SRE best practices nobody will care about where where in subscription models I feel like they have to follow a procedure because they have so many people subscribe and that will be I think. Better place to do this. Sorry, but it if it's only one big customer the most important customer.

Don't you can't bring any history best practice on the table when there's an issue there? You can't just want to survive. So I think to if you want to substance already you should have Should have good number of customers.

As if you rely if you're a small company or rely on one stream of income. I think it's hard to enforce these rules. and I think to to piggy back on that like, you know, if if you do it's a fine line right with a with a startup you've got like, you know, say one main customer you want to keep them happy because otherwise like that's that's who pays your bills but like at the same time How do you manage your customers' expectations from the get-go so that they basically don't walk all over you right?

So now we're talking about like different another factor that affects the success of your your SRE team, right? You've got like your your individual contributors on the SRE team. We've got management making sure that they they're bought in and you've got the external factor of you know, the customers trying to influence how your practices which I guess and to a certain extent then is is influencing Management in terms of how what they try to dictate for for your SRE team.

So then the question becomes like how how do you How do you train your customers to ensure your success and subsequently their success? Anna thoughts I'm thinking. That was it wasn't a quail as a statement.

It's like Anna you're thinking. I want to ride on this point a little because I think it is very interesting about the difference in models not to continue this line of discussion because I think it's not it's completely different thing but it does feel so much like we have crossed the Rubicon in some way in terms of like what is a software business anymore like Can anyone think of the last major company or last the hell just a startup in general where it's like. Oh, yeah.

No, our business is selling shrink wraps software to the Enterprise, right everyone at SAS. It's it's SAS all the way down. Which I think just mostly indicates the importance of actually thinking about a sorry because even if you are sort of listen to this and you work, maybe a more traditional, you know.

You know box and shrink wrap type of place like you probably have customers that want delivery over the Internet. They want this as a service. They want this description model.

They want more flexible terms and seats. I think you see that even some of the Star Wars like like jetbrains, right? Like look at jetbrains moving to a more subscription, you know, they make IDs sure but they are also moving to this very subscription heavy model and a lot of like Extra Value in online features look at things like GitHub.

Um Microsoft I guess which is moving towards like a certain more services revenue and You know, there's the markets changing and I think that SRE is and reliability more generally is just going to become a bigger and bigger part of how we consider. the software the software industry and I mean, I think also like the last three years I really shown us that too where it's like we kind of depended on technology and SAS for just about everything from like what is the status of covid and my city to I need groceries to I need to schedule like a vaccine appointment. Like a lot of people didn't have this like I depend on these things because they were going in person and with a world pandemic.

It's like, oh no like They're like technology can make those online but now all the sudden like I'm depending on something. You never even knew that I needed to depend on and yeah. And and even like major, you know, even I mean, I think that the the real probably wild story of the pandemic is just the amount of industries that nobody ever would have assumed would have like remote workers moving to remote work right like think about all the government.

Think about all the government employees around the world or even just in the US who? Need to interact with software systems all the time and suddenly we're doing from home and think about the regulatory concerns. Think about the security considerations.

Think about how much of a nightmare that sort of stuff is normally, you know, if the federal government can move to work from home then basically anyone can but that means that they need software delivered in a different way and they need and they will Boy, you think feder amps great. I wait till someone comes up with this SRE guidelines. It's gonna make everyone's life.

Super fun. You just making of terms Dawson. Yeah, mostly.

My micro influencer you're gonna get invited to be on the working group for those guidelines. I have it. I'm on too many working groups.

I would like to be on fewer working groups, actually. Thank you. Hey, they finally recommended that you stop doing password rotation, which means that it'll trickle down to the rest of the world and about 20 years.

What is that? Because password rotation doesn't actually like make things more secure. Immunity do three four to the end.

That's really good. Four five. Six.

Seven eight. It wasn't secure before rotating it right. Well, actually no, I think that the fully intangent zone now, I'm totally the I think the rationale's mostly the password rotation simply encourages you to record your passwords in an insecure way right like you write in a Post-It note.

Are you put in a text file or whatever because you get you can't remember what it is this 98 period and that means that it's truly easy for someone to go get it. Although as we've seen recently you can put all the recommendations in the world for security and safety and everything else and Someone will still put a root password and plain text the file and then your entire network will get owned by. teenagers yeah, there's a thread through some of the conversation that I think came up earlier.

We have about 10 minutes left as you'd be great to talk about when you step into a situation. It's not just I want to have an SRV group. Right?

Like I now have one and look at my shiny new SRE group. There's usually a there's a why why are we doing this? There's the purpose there's a set of problems that we're having, you know, the builds are not going.

Well, we're introducing too many bugs into production. We have performance issues. We have whatever it seems like isn't that one of the first questions you want to ask is like, okay.

What's what's important here? What's not working? And what's the most important thing to solve?

Is that a good question to start with? It's a good question your day with in general. Yes, great any job, right?

I also think because people get so fired up with these ideas. They tend to try and do it all at once as well which just and there's like the most common sense thing to say in the world yet. It seems to happen all the time.

Oh, so where it's where my observability cap for a second quite literally. Obviously, I'm biased here because it's it's a passion of mine, but I do think that one thing it's important. If you are looking at this is to pull out sort of the kind of through lines, right because a lot of most most of the time in most organizations.

The biggest problems all do have some commonality, right and quite often. What I personally see is that it's observability problem or lack thereof, right, you know. Or a very least it's very difficult to quantify any other problem.

You have that observability. So one thing to consider when you are saying like well we need SRE is asking yourself if you need SRE or if you just need us if you think you need us, right because you literally have no idea what's going on without and I think that that to me is the real benefit of an SRE or someone who's sort of there to ride her on the rest of the team and be like, well actually no, we aren't gonna just like FTP files to prod and we're not just gonna like randomly Cube cuddle apply, you know, crap from our desktop or our laptop. We we're going to do a process and I'm going to tell you what that process is and make sure you follow it because you obviously can't be trusted to do that yourself.

Um, I think it's just admitting that like technology is gone complex over the last 20 years and week just cannot keep on doing things like maintaining our our systems the way that we used to so we have to find a way to you know, bring bring the order to the chaos. It was it was cute before to be able to FTP into some log files. But like that's just not going to cut it anymore like It's just too complex.

I would actually challenge that point though. I don't think it was ever cute. I think people just did it because it made them feel smart fair enough fair enough.

Like I I loved it. Don't get me wrong. I had like dual 21 inch CRT monitors and I the green Matrix theme and yeah, yeah cranking.

Well, yeah, you know like you could press the goth and your hair move. Yeah, that was good stuff. But I like people always people do that stuff because they have to and unfortunately, they then think that To do it.

They have to do it that way right like it's this this awkward. Dance and like I think when when stuff like pipelines and like, you know policy Gates whether it's open policy or something else like those should be empowering for teams. If you've done them, right and if people are trying to sidestep them that is strong signal that you have done the wrong things.

We organization doesn't matter that it works somewhere else. Like at this group right now. That like that is caused friction.

And you know, it's it's the equivalent of the sticky note on the monitor to go back to like mandatory chapter James policies, I think. It does go back to that like business needs are going to be different for every single business whether they're like selling shrink wrap or they're selling Bender like SAS vendor like sorry subscriptions where it was that where it's like those gates are going to be very much different needs of the business and it all goes back to that question of like when you're starting an SRE practice, like what is it that you're trying to achieve? This is for the customers.

Is it the business goals? Is it because you literally can't write code without breaking production. Then and to add to that don't implement the Google way of doing things because that's what you think you should be doing.

The book was written by Google for Google not for you startup land, right? Yeah exactly. It's like everything there's like good little nuggets and and obviously take the Stuff that works for you, but you know, I think the common theme here has been like do what works for you.

Not everyone's running up the scale that Google is of course. Yeah, and I think that that needs to get highlighted in the book sometimes. Yeah, totally.

At every startup has written a global distributed lock service. No. the best chapters of the book If you haven't written two globally distributed log services for breakfast, then what do you even doing?

You're clearly not on it? Yes, sir. You're clearly not on that grind set.

I'm upgrading here. We got these versions. That sounds yeah fun.

So it's lovely. I mean, hey, at least you're doing it on gke and not something that isn't gke right. They're coming.

No comment. You know, I spent a lot of time this weekend playing around kubernetes and it's amazing to me is like actually no, no not not to derail again, but I think it's it is fascinating to me. How can Covenant like SRE is a defined job role has been with kubernetes because I remember when kubernetes was brand new and I remember seeing like version zero of like Kelsey's kubernetes the hard way and going through it and being like wow, this really sucks and we had built a kubernetes installer that I can't remember the name of I have to go look it up and get up but we build this installer that kind of Was meant for like installing kubernetes onto, you know, dedicated Hardware into like virtual machines or whatever.

This is pre gke pre everything. went through and it's amazing how much better manage kubernetes has gotten Until you need to do literally anything with it and then it's like oh this is actually 20 times worse than it ever was because this kubernetes has sprawled into being kind of all things for all people all possibly container orchestration scenarios, and it's like this is incredibly confusing and I have no clue what I'm doing anymore. So I think if you are a smaller Organization for whatever reason you started building in kubernetes, then you should probably hire an SRE just so you have someone that knows what the heck's going on with Kate's because I don't think anyone else does I would even take it to take a step back and ask yourself.

Do you need kubernetes? Yeah, I think it's too there's two minds of it. Right like this isn't an interesting.

Okay fun conversation for you ever everyone. If you meet up in person at some point find your biggest competitor find your find the person find a colleague at a company that isn't yours and Compare and contrast the engineering decisions that like started and where they've LED because I think you actually see really interesting. distinctions between groups of like this is the people that started building on kind of more these this is people that started like Monolith model of monolith scale up his foot Until It Breaks and then rebuild the world.

Then you have the people that were very very attached to a cloud particular Cloud. Then you people that kind of attach themselves to different sort of orchestration systems and you know other fun, you know, Fabric Technologies foundational Technologies. It's very interesting to kind of Trace that history with people in part, you know just in conversation and see like where like what broke for you that also broke for them.

But also, what did you avoid that they didn't and To bring it to the top of SRE. I think that's actually a really good practice for everyone that's in software reliability or just like this curious about how to build software like talk to your talk to people that aren't you like Take the opportunity to learn from other people about their like what went well for them. What didn't go well for them like share more we should be we should all be a lot more.

Should be friends more. I think that's a great advice. You know what we are.

We are out of town out of time. Yeah honest you kicked off the topic. I'd love for you to kind of wrap it up with.

Your parting thoughts either from the conversation if there's something you want to add that we hadn't gotten to. I think for me when you're starting to consider like hotter start your ass or your organization, like Don't just run because you are. Trying to reach for those reliability nines really try to understand the pain points of your organization and understand what your customers are needing.

Like, what is it that they're missing out on your experience like really sit down and interview them and try to make that customer Obsession translate into an experience that they're going to like fantastic Austin any parting thoughts before we wrap up. It's been I think this is a great job discussion. We I'm really excited to see what everyone out there on the internet.

Thanks. Be sure to drop us a comment or like it's argue with me about how we should actually all be enemies or something. I don't know but I think this is a good discussion and I think that it's these kind of discussions that help.

Advance the the industry so In a way, this was fulfilling what you described sharing our experiences and our knowledge. Exactly also, yeah and to go about and to really bring a full circle if please don't just throw your sres or devops people into service ownership with no no life vest and no room Book and know nothing because boy that seems like a really common occurrence and doesn't feel very good kind of sucks. All right with that we'll wrap things up.

By the way Jason. I think they called in the the Phoenix project Brent was the person who was the rock star that was FTP into production right if I recall, so, all right. Thank you everybody for joining us Austin.

It's always great to coast and I think for you for for germinating bringing this idea to life for our conversation today, and thanks for all of our guests. Thanks for joining us. We'll have another episode of the SRE show up soon.

So please check back. Thanks for being here with us.