Digital Transformation and the Art of Taming Feral DevOps – Techstrong Con 2023

Hello, welcome to digital transformation The Art of taming feral devops. I'm Paige Cruz a former site reliability engineer a forever devops Advocate and current senior developer Advocate over at kronosphere. Today, I'm really excited to talk to you about taming feral devops.

We're gonna cover the vision where we all want to end up with the end of our digital transformation. And I'll introduce devops a guide philosophy set of practices. You can introduce to Aid you on this journey?

We'll dive into when devops goes feral how to spot it and how it impacts you in the business. Which complements the next section tactics for taming feral devops, of course your mileage may vary it all depends on your org your Tech stack your people, but I'll give you some good starting points and we'll finally wrapped up with are we transformed yet? so the vision we are all sold at this point on the benefits of digital transformation increased efficiency greater business agility introducing new features as fast as your customers need them before they even know to ask and ultimately unlocking of new value for employees being a great workplace to work that's effective customers reaping the benefits of all of those new features and stability updates you're able to push out quickly and shareholders.

But how do you approach this type of fundamental shift? Ask most people in Tech and they will say devops for the past decade organizations have adopted this thing called devops when starting their digital transformation Journey. So what is devops exactly?

Like any Tech term it depends on who you ask all presented definition that rolls up devops into bringing together people processes in technology. And that order is very specific. You want to start with the people it is all about your people first your employees, then you can look at efficient and scalable processes.

And of course the tooling to support it all once you know what you need. If you ask the founder one of the founders of the devops movement how to define devops. He will say the most wrong definition one can give is that devops is a buzzword and I agree with Chris.

That's why we're going to talk about people first then processes in Tech. So at this point, you know that your business needs to become more agile, you know that you need to start this digital transformation. You've heard Whispers of this thing called devops and how it's worked for these other companies and whether you're a leader an enterprising manager or a dedicated I see you've decided it is time to do the devops.

and that leads us to when devops goes wrong feral devops again, just like with the benefits of the digital transformation. We sell devops on the end State this end state where you have connected employees. You have great collaboration communication is Flowing across boundaries teams departments.

Everyone's sharing goal and there's an eye for continuous Improvement. Sounds great. And then you look around after you've started this initiative.

Maybe you're a quarter in and you see that your teams are looking stressed. You see Engineers that are burned out tired. Some people are raising the flag for help things are people are demotivated.

And then you start to notice the business impacts. That changes are difficult to make whether that's applying security patches or trying to introduce a new feature. You promise customers two months ago.

What are your engineers doing with their time? It seems like they're just fighting fires and cleaning up messes left and right giving them no time to dedicate to the strategy of your business and making themselves unhappy in the process. So you may think well, I'll hire a consultant.

I'll hire my way out of this. I'll hire a devops team or maybe I'll buy my way out of this this new tool I've been sold is the way to see engineering performance metrics. Well, I'm here to say you can't necessarily hire or buy your way out of this process because right now you're in feral devops devops gone astray, you've got to focus on the people.

You've got at hand. So some signs of feral devops include a draining on call whether it's a too small rotation too many services not right-sized for the team or frequent unactionable alerts that page engineers and wake them up only to self-resolve within the second. Nothing is more frustrating than that.

We'll talk about learned helplessness a little bit later, which is going to be a barrier many folks will run into. Of course an extreme cases like mine. There's burnout.

I absolutely burned out because of on-call production pain being on a two small team. It just got to be too much to hold the world on my shoulders like Atlas so I came over to marketing where I still get to talk about devops, but I can't imagine going back to engineering. We don't want that for your team.

Other signs include information silos the development team has their own tools their own processes and they really just see their output as input into devops in the pipeline. And once once they write the code and merge the pr It is in Ops his hands now. Yes that wall still exists between Dev and Ops.

Sometimes you see this come out and really snippy and adversarial incident reviews, especially when things go wrong in a complicated system with a lot of overworked people. They don't have a lot of Grace to be kind especially if folks are playing the blame game. So be aware for that.

And finally hiring a devops engineer. Devops is not a single job title that one person can manage for a whole organization. That would be like saying they're a teamwork engineer.

They're collaboration engineer. It sounds funny right because they're actually a software engineer or an operator. So these are some things that you can notice to see if you are experiencing feral devops.

So what gives? How did you get into this state? We had the best of intentions so many companies have gone through this before surely.

We couldn't have gone too far off the off the path. you've poured investments into this change and instead of getting just a few of the benefits. You've actually seem to make things worse than they were before because you've told people we're on this change maybe a few of them got excited and they look around and see things deteriorating.

Well, welcome to the J curve. This is a phenomenon where organizations who are on their Journey for Transformations tend to exhibit early success. When all the eyes are on the project things are getting started people.

Have that vision of the future. So Crystal Clear in their heads. This is followed by a period of diminished returns, which is where we are the trough of the J.

That's below the line that is feral devops territory for sure. But what we have found over time is if your organization charges on. Treats this as the spirit of continuous Improvement.

Not just a single Destiny destination on your journey, but a new way of life for your company. then not only do you experience renewed levels of productivity and achievement but also sustained So what we're really seeing here is it takes time to learn it'll get a little bit worse before it gets better because you're trying new things. You're trying a new paradigm way of working and be prepared for this now that you know, you can be aware not only of the signs but that acknowledging that this is a natural part.

of introducing change management So be prepared for the J curve. Don't let it catch you by surprise. Something else that comes into play is leadership's bias towards optimism and this one comes from the 2022 state of availability report from moogsoft.

They're study. They studied and showed that the gap between leaders perceptions and the experience of Engineers and ICS on the ground. Was wide.

That leadership exhibits this biased towards positively reporting capabilities. Whether that is management believing that they've got solid cicd in place because they bought the tool and it was rolled out. There are many ways that this disconnect arises.

But the result is management has one idea of the experience of employees their expectations for efficiency based on what they're hearing and they're building plans in the future on top of that. faulty Foundation then you have engineers and ICS on the ground toiling away and trudge fighting these fires not getting fulfillment from their jobs because they're working on putting out fire over here instead of really cool career defining feature that can make the company money, which is what they want to be doing. And so this consistent difference in perspective the longer it goes on the longer.

You're going to stay in that J curve of disappointment in feral devops. So it's important to acknowledge as leaders. What might you be missing and how?

might you be getting Overly positive feedback from your management or your teams take a look at that? So double clicking into learned helplessness if this is a new phrase for you like it was for me. Here's some signs that you can listen for in your org.

This is how we were told to work. We just don't have the resources for that. That's great, but it would never work here.

This is indicative of an environment where Engineers understand that the tooling is inadequate and imprecise. They understand the hot spots in the areas where their architecture is holding them back or they know the landmines of technical debt that exist out there and would love. To be able to neutralize them.

But feel the pressure to add new things. So you have engineers? That working in the system.

They've learned instead over time through the cultures of leadership in the organization that deadlines delivery and Reporting are more important than their actual productivity or the quality of their work. What this leads to and the impact for you in the business, especially on this journey of change is that these folks actively resist change even when? Optimizations like devops improving the clarity of communications revisiting your incident workflow.

Whatever it is is suggested. Because they just don't believe based on their past true historical experiences with the company that they have the will to change that they will invest proper time and attention building in buffers to commit to change not in not being devops theater, but to actually take a real real crack at change management. so when you hear these things.

And you're in that J curve? No that these are the folks. You've got to get on your side to become Advocates.

You've got to understand of course where they're coming from why they believe that and over time show that you've got the will to change and you're willing to work with them, but this is one to definitely be prepared for the longer you let this feral devops phase. Stick around in your org. The risk that you have is these folks who have learned helplessness turn to burn out.

And that's what happened with me. I spent many years as at startups as a site reliability engineer was a Tiny But Mighty teams. We tended to be what I call kitchen sink SRE so we would do a little it little security all of the cicd pipelines and infrastructure.

We did the technical infrastructure for the orchestrators the cloud the networking. It was a whole grab bag of things and this illustration. is is kind of what it felt like at times it felt like everybody around us was saying you build it you run it we've made you know devops is so great.

It's nothing's like it used to be 10 years ago the future so bright and shiny and yet as an operator the buck stops with us. If a service is unowned in your system or no development team is willing to is able to put in the time or the maintenance. Well what happens when you've got to update a library because a critical severity just dropped.

A lot of times that falls to Ops and this is the type of toil and drudgery that wears on you over time. and so it from the perspective of an operations engineer. I do feel like we've got a little bit of unequal responsibilities between Devin Ops and this is based on my own experiences and past organizations your organization.

You may find out that the developers are feeling more stressed that they have gotten more responsibilities in that they they would like to share more with operations. The important part is that you look at your org your people and your system and your business. So I mentioned the risk of letting learned helplessness Fester is that it turns into burnout.

And this is a snippet from the chronosphere cloud native complexity report that shows this Gap. We've been talking about between the leadership and director experience and perception of the business. versus the individual contributor and engineer just look at the difference in those top two numbers 45% of Engineers say that they are frequently stressed compared to 20% of directors.

Now that stress is certainly not equally distributed. and the most sobering statistic for me. Is this 9% of Engineers that say that the time they spend troubleshooting actively disrupts their personal life.

This is not oh I'm on call and I know to expect one one week out of every two months that I've got to carry a backpack with my laptop and I can't go to the movies or whatever, but this is I'm trying to do my job. And it is cutting into my dinner time. It's cutting into my time to go to kids soccer games or for me to go to an alpaca Festival.

It's interrupting their personal life and I can say this is not a phase that many employees can stay in for very long. So this is the acute case of when Ferrell devops is allowed to persist for a long time. So, how do we tame feral devops?

Let's think about a scenario. Maybe your marketing team. Hey wants some shiny new features that they can brag about at the next big conference.

But over in sales, they're seeing really high customer churn because the product has a bunch of technical debt customer success is reporting unhappy customers because they would like a new API endpoint and all of these teams have competing. All of these teams have different lenses and perspectives on the system. But ultimately, we all want to be solving and driving towards the same outcomes in solving problems for customers So, how can we in a post devops World handle what used to be teams talking over past each other playing these games of politics?

Well, I will tell you. It starts with the teams. I think I'll say it one more time.

If there's anything you take away from this it all starts with your people people first. Open up those channels of communications because really this change is important to start with the teams. They have to make their work visible their toil visible the the burden of on call Visible to management.

They got to give you that data to help close that perception Gap. Then it's on you as Leaders to identify those gaps and rank them. to discover improvements to make sure that there is time in the roadmap to invest not only in future features, but also ongoing maintenance and sometimes that means throttling the amount of change or net new things that you're introducing it all depends on your org your people your Tech.

Once you've identified those gaps and you've started to work on closing them. It's all about iteration. Because really to me devops is about introducing a feedback loop of continuous Improvement.

It is not necessarily. Clanting the flag saying on this date, we achieved devops and the business grew by two million percent or whatever. It's really about.

Hey, we tried this thing last week. It didn't really work out. Here's what we learned.

And now we'll try at a new way. It's about being open to change not only of your practices, but even how you get your job done because again, this is a paradigm shift. We're not naturally used to working across teams and working with folks who sit outside of our discipline.

so this Playbook of getting the people talking first to you to each other across teams figuring out where for your particular company the gaps are whether that's knowledge training. or people so that's our Playbook. So we'll go back to our image of that poor devops team that I had been on where everyone's asking me to deliver things.

That's not what we want. We are driving towards this outcome. of easy changes happy retained fulfilled employees delighted customers who want to buy more of your products because you keep coming out with awesome stuff.

And you've heard it here you can start here and now it is very easy to get conversations starting to get people talking to each other. Creating those feedback loops and the best part is you join a very large community of devops practitioners, whether you want to go to a local Regional devops stays conference, whether you want to blog about your experiences with devops or share them in other kinds of forums or talks. The whole Spirit of devops is sharing what we learned with each other to make everything.

Better because the world runs on technology and this happy glorious end-state shouldn't just be a Faraway Vision that we never realize it's something that you can achieve today being mindful of the J curve and keeping focused on connecting your people your employees to do open collaboration communication sharing goals. And of course continuous Improvement. Thank you very much.

I am Paige Cruz. You can find me on the internet. My handle is pagerduty, and I would love to hear about the struggles or roadblocks.

You've run into when experiencing this. I'd love to hear how you got yourself out of the J curve. Feel free to drop me a line, and I can't wait to connect.

Thank you.