SLOconf – Kit Merker, Nobl9

This is texturung TV. Hey there, this is Kip merker. And we're on Textron TV.

This is kind of kind of cool. And very fortunate today I get to talk about slow calm and if you're not familiar with lowcon, it's a virtual event all about service level objectives. Bring it together practitioners from all over the world to talk about reliability engineering platform engineering devops and cloud.

And I've got a great group of people here today and we're gonna talk about what's going on. com, it's me 15 through 18th at 10 while you work totally virtual and gonna be a lot of fun. All right, so I got a great panel here and I just want to jump right into it.

So Greg you you're talking at so comp about the hidden taxes. Can you tell us what is that talk about and like why are you why are you talking about? Yeah, sure.

Hey everyone. I'm Greg Arnett. I'm co-founder and chief product officer for cloud truth were seed stage project getting started around Reinventing configuration management and my talk is around this concept of the Hidden config tax.

That's affecting your uptime SLO and we're shining a light on how misconfigurations lead to unplanned downtime which affects uptime slos security incidences and generally grinds to a halt fast deployment processes and cycles that we see our our customers really trying to get to So the the Talk's going to dive into the common causes of misconfigurations and discuss seven practical ways that teams can get around these issues go faster and overall increase their release velocity and uptime so looking forward to this discussion. Well, I like how you said shine light on which is one of my favorite acronyms SLO shine light on. I think that's a great one.

So the nice job. I'm trying to get that to catch on. The certainty so you work actually on a different SL kind of project you're at meta right and you work on flick.

What is the cool name? What's the what's click all about? Yes, exactly.

Yeah. Hello everyone. My name is Sergey.

I work on winter confident. So yeah, basically, I don't even know what this name stands for to be honest. I just a cool name, but the the project overall and like the problem that we're solving is trying to bring standard way of measuring it's a loss of the company in trying to understand things backed off a solo violations.

So I'm going to present with my colleague David through who is from the same team as I am. And the talk is going to be about acceler reviews about our journey of when traditional abuse so like a quick outline what we're gonna discuss and we're going to talk about is basically so let's say we have a metric. Let's say we have an SLO and say let's say we have a way to actually look at this lost course, but the question is and the challenge is to make this actionable to make teams to actually start getting value out of it to to be able to react on the Excel evaluations.

It's Etc. And that is how we trying to approach the problem bringing this reviews in allowing a team like bringing the process around. previewing Castle also making like tracking of violations and making able to action like file follow-up tasks for for any or any issues detected and yeah, basically bringing this like single and concise experience for users to actually be enable to to get insightful information about the about decimals.

What do you think? Oasmere talks, or why people come check it out. Yeah, I think I think the main takeaway is basically an approach to the challenge in in general because like you know that technology is not a problem in many cases, right the people that's like the domain the main thing that we need to focus on and basically our talk is about how they actually approach the problem what we did in order to encourage him to go and to review the slos.

Yeah. I think that that one of the most interesting takeaways. Yeah.

Well speaking of people so Francis, I'm not taking you're gonna talk about happy and you know software would be easy to work for the customer, right? So what about your talk? Yeah, definitely.

Hello everyone. My name is Francis gel Perez a product manager worked with the Salesforce neurology AWS Oracle and it really a passionated. Product manager care about building running reliable services and how we really measure our service well as all to achieve customer happiness, so it might have I really like to keep you said about the customer happiness and how we're measure it and how we leverage as all to help us to pursue this journey.

So in my talk, I will really cover. Why not how we Define our so and also people dive into difference between monitoring and observability and how as they all have a change these industry and how we really bring ourselves to Next Level and I really excited looking forward to discuss these and also we'll touch on service ownership devops and a white so important also the care about it. Well, it's interesting you come from the angle of product management.

I think we hear a lot about reliability Engineers Otis. Do you think a lot of product managers need more exposure to slos? Absolutely.

Absolutely. You know one thing we have been talking a lot about the area service ownership. Service owner is a managers engineer.

You notice everybody who owns a business who owns a service product manager Devin and the one of the most important that personas opposed to care about as old because number ones as PM you are absolutely have to understand your customer journey and how you measure a customer Journey how you make sure bring, you know expectation to Commitment back to your customer and PMs should really care and not only care but also clearly Define how we Define our So based on customer experiences. One last but not least Amaya from AWS and you know you're coming back for I think the I think the third year that AWS has been a top level what we call official sponsor at slowcraft. But thank you so much for your support.

Maybe you can just tell us a little bit about you know, why why you're getting involved in slow conf again and what you're expecting this year. Sure kid. First of all, thank you for having me here.

This is really amazing. Hello everyone. My name is imaya.

My full name is emic monitor now then so I tried to shot and it's the make it easy for you all. I'm a principal solution architect. I work on observability and I'm part of the business development organization in my company and we have a lot of Specialists and part of the worldwide specialist organization that Focus specifically on ability, right absolutely say it's a critical area for a lot of customers on AWS in particular.

We talk to customers all the time customers different sizes doing different. Things running a lot of exciting workloads on AWS. And one thing that is very common for all customers is that they always come and ask us like, how can I best serve my customer by making my service more reliable perform much better and also be more reliable, right?

And the the only solution that is to you know, basically be very intentional about setting up observability in your environment be very conscious about like what you're measuring and how you're measuring and to do all of that. It's really important to to Define slos as those give you the direction the specific area that you need to focus on and with that you can identify slis and then automatically you know, what exactly you need to collect and what you need to monitor. They're also if you look at it a lot of customers these days in particular focus on cutting costs and being more efficient and it's not that they only have to do now, but it's that's been Something that they have to do all the time and but there's been a renewed focus on being more efficient this year.

So defining slos making being very intentional means you're cutting a lot of fat out of your monitoring solution itself. You're probably know exactly what kind of signals are actually adding value in your monitoring environment and probably don't necessarily collect those things that you don't have value. So I'm really excited about this and really excited to see what everyone else has to say.

I'm looking forward to the conversation and the conference, okay. Awesome. Thank you so much.

Well, I want to I want to start digging in a little bit and one of the things that I've heard a lot is, you know, hey, I work at a company. That's not Google. It's not that Salesforce.

It's not Amazon. It's not meta and I hear about this that's a low stuff. But you know, is it really gonna scale down to my company or the way we do things and things like that and actually your take on this because you know, basically like why adopting a close so hard why is it so important?

And what would you like me to somebody kind of coming into this and maybe isn't working for one of the big, you know, I get this main companies now, I don't know mango. They got some new acronym that you speak thing. But what's your take on that?

Yeah, I think I think it's very good question. Actually, I use the different companies before meta and I don't think there we had like a strong sloculture like, you know, establishes local culture. I think it's it's common.

Unfortunately that the more customers you have them like the clearer expectations. You have the customers the the more like value you put into the into the slos in general. I think it all comes from customers and basically as soon as you provide the service you you want to to guarantee the service, right?

I think like as for the As for the As for the smaller companies, I think it's getting though. I think it's getting because like you might see like different like smaller smaller companies. They're bringing out of the book solutions for for other companies and I think like your company also also doing similar job, right?

So thank you for for doing that and making an impact of on and that thank you. Yeah. Yeah.

It's a good point. And actually I'm here take on it. Are you the role at least that's a lot of that who you see adopting or eating other kind of teams that are kind of looking at the world or do you have to take on that?

Yeah, so we did a survey of 900 devops and platform engineering leaders as part of sort of the product Market fit Journey behind my project as well as getting in touch with really what concerns teams are dealing with these days. And so we saw this interesting intersection and it ties into your question, which is there's this movement new viewing devops and platform engineering teams. And that's something that we're not going to touch right now because it's not a whole bigger subject but that combined Universe of those folks and those particular roles are being tasked by their product leaders to figure out a way to implement code and systems with the ability to implement slos and what we found the intersection with what we're working on an slo's is around this kind of You know hidden issue this blind spot that almost every team has as they're trying to scale getting their config under control was really working against an SLO an uptime SLO.

So it's coming from the product side of the org is from what we saw found in our survey questions, maybe in larger companies that's coming from engineering leadership. But from the sort of mid-sized companies organizations with a couple hundred Folks at two thousand. It's product leaders that are really kind of introducing this concept to the engineering teams.

And I think they're getting kind of a full range of results in terms of pushback to embracing it because everyone can now rally around a common set of patterns and metrics that all is in the right direction of you know, driving customer value. Whatever way each organization is doing that whether it's you know more SAS customers more, you know, Consulting clients or the range of Gambit in between. So it's really seeing it coming from the product side of of the customers base.

So it's really interesting that I saying earlier Francis. Do you have anything to add? Yeah, absolutely.

I'm just really happy to hear that in my opinion regardless. Big business Enterprise. So I SMB smaller.

This is really about the customer Journey. You know, how we make sure not only building, you know cool features, but also make sure your service reliable and because the end of the day your customer journey is your business, right? So I'm also very happy to hear that and now the only Engineers because all personas we all serves owners, you know part of the management.

In the business owner, we all have to care about that. So and how we really clearly understand is a journey and it defines and help us to launching product based on our commitment to our customers. Yes, very exciting to hear that.

Amaya I want to come to you on on kind of related topic but you know, I think we look at operational models for how to you know, how to run large scale operations Amazon hasn't necessarily been a huge advocate as a that's a little company but it's you know the recent year. I know you and other people on the teams that they do that the man how are you thinking about it different right that there's different ways to an operations. Are you fully bought in the air budget to the way you've seen that with customers or do you think it's one of many approaches that can be taken?

I'm actually really Well, definitely. Yes, like I mean, it's it's there more and more customers who are interested in in setting up SLS and really identifying how they can sort of set up like everybody just like you mentioned right like to really find out. What whether they're doing is actually meaningful or not.

Like it's really important that we're right. So you many customers that I talk to many that they're using let's say, you know some platform acts and they want to migrate a platform. Why the reason is it costing me too much right and maybe there is an optimization opportunity available there, right?

So whether you go to platform why but maybe you might feel like you're paying too much or maybe you will actually be end up ending up paying more than platform acts because you are not being intentional. So then going back to the same intentional being intentional and setting up measurements and all that. We see a lot of customers do actually come and say that they want to reduce costs and one of the one of the best way to do that is to you know, setting up as solos and being very conscious about like one day monitoring and really ends up being the best way going forward and also very I think the best word there is is kind of more reliable and consistent manner to approaching So let me so you mentioned cost.

I want to talk about any different thing costs productivity. Okay, and this is something that I I care a lot about is, you know, keeping things productive. I think there's a direct correlations the cost, you know your staff and your team and how much they're able to produce.

So when we think about productivity for engineers, how do slos make a difference I love to start with turkey on that and then we can go around and get some other answers. But how do that's close make a difference for productivity. And what is the pain point and it might my favorite one which is, you know, reducing noise alert the three in the morning, but how do you think about sloth in the context of making engineer for productive?

Yeah, that sounds interesting. I think there are different different, you know. Aspects in the question that we that you can touch properly like the first the first topic is pretty much.

So whenever you have SLS, you can actually use this information from slos to let's say reduce the alertness. It's one of the like videos on Colbert now at the end of the day, that's one of the one of the things that we seen as meta how slos actually helped seems to to improve their practices. Basically improving reliability in the long term right that also helps to like improve the system make it stable and basically lead to the same goal, but make make a call.

Like more or less better. Another thing that I would like to mention here is slightly slightly an opposite side, like for example bringing, you know. Isolated views as one of the examples like in the total we're going to talk about bring some at some some of the trade-offs into the process because whenever you introduce this whole reviews, it actually takes some time from the teams to do a solar review rather than focus on engineering for or on any other type of work and the important thing here.

The important exercise here is to find the balance between processes establishing the process and follow the process of reviewing slos and also like doing the work because yeah, it's there is like a competition here. So tuning Place making sure that the teams actually get well out of it and changing it over time. For example, it's one of the most important things in the journey and I think as soon as we have this clear we can definitely benefit from from from using slos and and make scene like more productive in the long term.

He made a really interesting point about tuning out to Lowe's over time, and I'm curious maybe Greg you haven't I'd love to hear your take on on. You know productivity from sloth as well this tuning idea. Like can you set you know a perfect look and you said enough slow once and have it be right forever or obviously, that's not the case.

But what's the problem there? Well tied into the productivity productivity theme is this concept of developer toil. I mean, they kind of their two sides of the same coin and that's significant survey that we ran.

It really uncovered this hunger for understanding the impact that this developer toil has on overall release velocity. And again, the end goal is getting new things in front of your customers and whatever way you can do that. Whatever.

The new thing is, we often think of it's a new SAS feature or some new API or something and so the perspective that I'm kind of taking through my discussion is that this whole concept of managing secrets and config at scale is an area that needs a bright light shining on it in terms of educating folks on what this impact is having on your release velocity. So we've seen examples of customers that use these seven Factor config principles increasing their release velocity by three x and that's Huge gain in terms of wins time to wow in front of your customers. And the way it kind of ties back to the productivity theme is everyone's got a hunger for adopting good patterns and practices and what we found is that especially with this fast Rise of kubernetes and containers and moving from monoliths to microservices and small teams, you know working in parallel Loosely coordinated each of those teens were making slightly different choices around how to manage this complexity and that leads to in number of patterns that then the devops team or the platform team or whatever your centralized team that has to take all this new functionality and now deploy it to a production environment.

They're the ones that are now faced with having to manage that teammate did config and this pattern Team B did a slightly different way that increases the time the troubleshoot outages resolve issues. So that kind of all works for undermining and slos that are around up time and resiliency and reliability. So anytime a team can adopt patterns that have been kind of vetted and seem seemed easy to adopt across all teams are going to have big wins in terms of reducing toil.

I see adopting SLO patterns as a key key initiative in that direction. So Maya how come the cloud doesn't get rid of all that? Well, what's up with that?

Well, well it is it is common. Like it doesn't matter whether you use cloud or on-prem. It's really doesn't matter.

Right so end of the day you have to have this strong, you know devops practices that you need to follow and I think what Sega made a great point before about reducing the number of alarms, which is really really important right? So which as long as you are optimizing each and every team members workload in this case reducing alarm. So in that case, then people are not notified to just go jump on something that's really irrelevant that really saves a lot of time that really improves productivity whether you run it on the cloud and not on the cloud.

It really doesn't matter the day. It's about what improving improving productivity improving performance of every single team member so, you know, whether they're learning. I don't know.

So it's a Francis as a product manager how you know easier for being a product manager. I have a product manager background too. But you know, it's probably the common misconception you want to always be shipping features, right?

The features are the most important thing new capabilities new innovation. At the same time. Our customers needed reliability.

Our users care a lot about reliability. How have you seen that play out, you know keeping your engineers productive building the right things balancing the reliability versus future State maybe walk us through like a how that discussion goes and in your in your experience. Yeah, absolutely since we have been talking about the product PVP before we talk about these achieved I also want and you know folks have been talking about here is about engineer process.

So one thing I really want to emphasize here is all help us and have only reducing noisy alerts, but also make sure those actionable alerts on the right kind of approach. For example, if you're gonna wake up your engineer three am You want to recap your engineer if it's okay customer on your web page? I want other something the pages and downloading definitely but if you have Alert on your climate change signals, for example, hey, your CPU usage probably reaching certain, you know percentage.

You may not want to wake up your engineering three. Am you know, all of those are really so help us to alert on the right signal so wages really important a concept here, you know here you didn't pay me to say that and it's really amazing novel night a solution. Give us a very easy user solution to help us to send those up.

The second upon here is like you mentioned that we want talk about the tree dogs, which is also related to engineer productivity. You know in an entire product launching process. When you try to avoid with you know, all engineer organizations are changes.

Oh, you know folks are working on the project a suddenly. You have some urgent that you know firefighting coming. Let's really quickly change priority.

Maybe today you have difficulties tomorrow you have to fix that which is really the process-wise one thing you want to avoid cheetah of here is How much time you budgeted for new feature launching how much you really should focus on productivity in reliability availability those type of work. I think in product management every single time only do our yearly quarterly planning leverage your soul. Make sure you understand.

Where's your customer Journey where you are with your service how happy your customer again? Customer happens how happy your customer? How you budget how you really want to carve all the time for reliability work and to answer your questions really about the depend upon the age of your service and you really want to balance and you can round another thing.

I want to emphasize as a product management is best practices you have to do both. if you don't do a new feature launching You lose your commitment your customer. If you don't work on your reliability, you're not keep your commitment, either.

You have to have a good balance. When you plan your roadmap priorities regardless is codular planning yearly planning again depend upon stage of your service and how stable and how reliable how happy your customer is balance letter, which really helps us to bring in productivity to engineers. I think that's music the most Engineers here to hear a product manager say that because I think too often we see more of a fight or maybe a negotiation right where reliability work is the bit of a trade-off there.

So let's talk about about slow comp the event. Okay, here we are. It's you know, 2023 and people are back to going to in person events and everything and one of the things people have asked me a lot about well two things about Socom wanted have a very set everything there is to say about Hello by now and turns out that that's not the case.

Yeah, I think 70 talk coming for flow comp this year, which is just unbelievable and second people have said well, you know, we want to get back going to events in person and I think one of the advantages we get confidence back that the global Community we're all joining and we're able to do it while we're working. There's no travel no travel expense no time off of work etc. I don't care with me.

You know what your thoughts are on how events are evolving how we're learning about things our building community in this sort of Host pandemic World, I'll just go around the table here. So maybe Greg give me a few thoughts on this and then we'll go around see what everybody have to say. Yeah, it's really timely to bring this up.

We're mapping out our remaining 2023 event budgeting and we're indexing more on Virtual events right now just from a point of view of Team productivity and budget and so forth. I think virtual events. We see the power of them since the pandemic Force just into that mode and you're totally a spot on in terms of it.

Let's the most people possible participate in the content at their own pace. And so I love the way that the slow conf is actually being run this year it comparison to some other events that were virtual that went completely in person and a lot of folks weren't able to attend those because of various travel restrictions budget restrictions. So we're not quite back to the pre-pandemic everything in person.

I think there'll be a nice healthy mix of virtual and hybrid and in person going forward but virtual for us is the way that we're thinking for the rest of 2023 in terms of how we spend our time and which events are going to be the areas that we get involved with interesting Sergey. What do you think? Yeah, I absolutely regret.

I think like basically the combination of two mods is kind of it sounds very interesting. Like my personal take here is basically the virtual versus versus person. They basically optimize for different for different goals.

Well, like virtual one is optimized for learning the the onsite might be optimized for like networking connections such as a transfer. So I think combining this to most make sense. So yeah.

Yeah, Amaya. Yeah. Well, I think Bots again Greg made great points there while I like in person events.

Like I think it likes like I mentioned it encourages more networking. I'm I myself more of need to really get better at that's I'm trying to develop my networking skills. So I would like to add more networking opportunities like in person events the but one good thing that came out of the pandemic is that we all learned how to work remotely and be more efficient while doing so and I think that just that is definitely a skill that we are all going to keep using again and again in the upcoming future.

I think men there are many conferences that are there that is that a remote that are online and they actually give you the flexibility of you learning when you want and how you want. I think that's a huge advantage. Same time.

Add AWS. Also we did conduct reinvent this 2022 and 2021 in person and 2020 was was remote. So and we do have Summits that are in person.

But we also do virtual conferences as well. So some it's not all Summits are in person there. It's it's a blend.

So I think that the blend is something that's going to stay with us for a while, I guess. I'm coming to the New York Summit in July can't wait and reinvent. Of course every year in Vegas.

Can't wait Sergey said a comment about the virtual Venture optimized for learning Francis. Do you have about final word on Virtual events? What do you think about that?

I put my part of the manager head on so I I, you know, launching running products similar to lunch and learning on the event right in my opinion. So first things we understand who are the audience, you know, Persona and the job to be now who really care about these what kind of problem value with brain by these type of events. And if you look at our Persona English space is very exciting lots of devil, you know, practitioner and product management given that all of busy schedule and I think virtual event works.

Well, for example myself all the you know, previous years local sessions really exciting the last seven knowledge really and also allow us to share. For example when I say who this is a whole organization Hey everybody, check out these right and I sent to social media. But again Pro and account.

What we don't have is kill you. Remember you and I met a few months ago. I was presenting at reinvent in Las Vegas and that we don't have that in person, you know fun, right you really get together a group of folks everybody exciting talk about this stuff.

I think in the future probably we could do hybrid, you know, get the, you know, some local community get together by the ways, you should take advantage of virtual event so easy for us to really relation check out the content easy. Yeah, that's right. There's there's no replacement for in-person.

We have found that-- you do if you don't try to replace an in-person event with a virtual event, if you treat it as its own, you know, its own thing and you design it here both your points right? Like all your point. Really you want to optimize the right kind of outcome and I think that really is what makes the difference and Yeah, it's the interesting to be able to have this is a new tool in the tool about what you know what I mean from what we've learned through pandemic.

What work what didn't work. I remember something that I attended that were, you know vertical boots that we're not exactly know. It's exciting thing.

But yeah, great. Okay. So I think we're we're running up a bit on time.

So we'll just wrap up here, but I can just have you each of you say like this maybe a couple word about why somebody to join flop on the fence about it. I haven't registered yet and thinking about it. Why should I register right?

Yeah, it's it's gonna be a great library of presentations on this important topic and whether you can attend the actual live unveiling of this or have access to the recorded artifacts. I think that's the one of the really cool things about these virtual events is everything's recorded and you get the index and you can share the links with team members. and just the more Shining Light we can put on slo's is the key here.

I love it Sergey. Yeah, I think yeah, I think basically we all like share the same Community. Right?

And this is like one of the ways how to contribute how to get long village from the from the community. So basically, yeah, you can join the conference to get like the best practices to see what other people in this you're doing and use this information to actually get your own improvements in these laws were at your company so became like Matter like the subject matter expert the company. That's good point Francis.

So three is a good numbers standing in additional shoes three things. I want to emphasize number one join a community. You know, where is the changing as those important?

We all running Services things are changing because all your resource of dynamic you have to care about this. So number one join a community number two, we have lots of great content a good speakers everybody put a lot to effort with a huge energy. So let's have a good discussion, you know join the community have good discussion three have fun together and we really really want everybody join and we can have this together because for sure it is going to happen again next year.

So that's really do this together. Super exciting. Thanks and Amaya.

Well, yeah, I think this is going to be a great event for anybody who wants to learn about slos and how the how it's actually being used in real world because there's a lot of practitioners for for speaking in this conference. I think it'll be really important for anybody who wants to get into this space are trying to understand and optimize their environment really wanting to learn about SLO so they can really be efficient in their observability platforms should definitely watch out just hello conference. And nobody mentioned swag most important thing.

We all get so conf t-shirts with stickers. So you make sure you come for the track. Well, I want to thank the panel today it was super inviting and I'm really really even more excited for so confident.

I was a half an hour ago. If you're out there you haven't checked out so comp yet. com and register now for free you'll you can attend virtually while you work in engage with all the content it May 15 through 18 to have the schedule and everything else out there.

Make sure you sign up now you get updates on what's coming and with that strong TV worker keep growth officer. No line. Thank you so much and we're out.