Improve Automation to Reduce Toil | DevOps Onramp 2023

it Hi, thanks for coming to my session. This is improve automation to reduce toil. I am Mandy walls.

I am a devops Advocate at page or Duty. And if you'd like to get in touch with me, I'm lnx chk on social media. com.

I love to chat with folks about automation incident response and all that stuff. So if you have any questions, feel free to reach out. So I'm going to talk a bit about Automation and complex environments and all that good stuff.

But to sort of set the stage. I want to start with like a common task that someone might need to accomplish to get their work done. Right?

So this is our developer Alice and Alice works on a micro service. It's a feature that's part of a customer-facing application that her company runs when Alice or one of her team members have finished some amount of work whether it's a Sprint or a feature or a story or whatever it is. They're responsible for the first round of Sanity checks before the code goes into the big full build environment, right?

So just to ensure that her code is gonna work. She's going to request a sanity check environment from Cloud operations team to run in the development cloud account. Alice doesn't have direct access to the cloud account because it's restricted to a limited number of people in the organization for whatever reason there could be some history there, right?

There's probably a story. So she submits her request to the cloud operations team. It goes into a queue with requests from other teams, right and it'll be handled eventually maybe first in first out or maybe folks are like skimming through and picking out the same side types of jobs to do them in a batch or whatever the the team does.

The first subtask for her ticket though is to get approval from finops. Within Ops Team reviews the request tickets and looks around in the accounts to make sure that the team that's requesting the access or the additional resources isn't over budget or doesn't have too many environments already. If that's the case it will kick the ticket back right to the requester.

So if synops then approves Alice's ticket, it'll go down the the ticket queue the rest of the way to Cloud Ops at some point. Hopefully Alice will get her environment, but it feels really clunky right like we want to make sure all the boxes are checked. We want our security.

We want our financial responsibility in place there but like Alice is now waiting around for things to happen. So she can continue with her work and our Cloud Ops folks. I just continuously going through that ticket you like doing maybe the same tasks over and over and over again for maybe for different teams, but starts to feel really repetitive right and Really take all of this time like we started using the cloud.

Because it was fast, right we can get what we wanted really quickly and this no longer feels like that. Right but we have now complex requirements in our environments. We have compliance and security and financial guidelines.

We want to adhere to so we need to examine our workflows figure out where all the knowledge is and how we can smooth these processes a bit with some smart on emission. Ultimately. We want Alice to continue working through her workflow, right?

We want to get her to where she needs to go quickly so she can keep shipping features that our customers love. That's the whole point of doing all the work. So all the folks on our example have goals.

They're responsible for they have different products and services. They're permitted to have access to and most of all they have specific expertise for their roles. How much knowledge and expertise gets shared among different teams varies a lot from organization to organization.

If you've ever worked in a really small organization or a startup, you know, there's maybe one huge Tech Team or one. Combined Tech team will be huge right and maybe five or six people maybe centering people and they do everything. They've got the cloud stuff going and they've got the VPN running and they're some buddy over there is working on the telephones and the zoom accounts and all that kind of stuff but they're all sort of together and they all can reach in and touch the same things.

Right but as our organization grows and our needs get more specific and more complex than a lot of the work that has to get done has to start to be split up, right? And we find that in lots of places individual teams have sort of spheres of influence based on some kind of work that needs to get done and it requires specific expertise right in our fictional technical organization Alice and the rest of our Dev team. They're not permitted to directly provision Assets in the main Cloud accounts.

The organization has requirements around who's allowed to log in and deploy in the cloud and that is limited right now to just those Cloud Ops folks. And the clouds folks have lots of tools at their disposal. Maybe they wrote some in-house.

Maybe they came from the vendor. Maybe they downloaded them off the internet, you know and modify them to work, but they're after efficiently deploying components for all these various teams. But the tools still have to be run and managed by Cloud Ops and then we've introduced genops as we've been trying to you know control our clouds Ben.

We have this new functionality that we put in it's a new team kind of sits there in the middle and they have a little bit of access to cloud and a little bit of decision-making power, but they're not really in charge of the development team and there's that in charge of the cloud operations team. There's kind of an intermediary but these teams know their stuff right they have enough to work on though to keep them busy without Alice wanting to do more of what cloud Ops does right so unfortunately having important skills siled off a bit like this means it's challenging to keep speed going right we think about how work flows through the organization and it sometimes hits into these bottlenecks or little speed bumps where there's you know, some teams that are specialized or restricted for specific reasons like our Cloud Ops Team, right? So we want to get going with some automation here and we want to be able to delegate it effectively.

When we think about I want to automate this. What is automation right a lot of the work that needs to get done to get a piece of software up and running nothing to do with the application code itself, right? It's all the other stuff that has to happen.

So that our code has this nice save cozy place to live when it gets to production. Our Cloud off team knows the right spells and incantations to make all those things happen, right? They need to get it there the right way the right controls and the right environments.

I don't really want to do it themselves because after a while it can get kind of boring right but our application Engineers also don't have enough time to become very well versed and all the intricacies of the cloud that we've chosen in order to sort of be certified to be allowed to touch the cloud environments. So we have like this disconnect, right? We want to be able to facilitate a better workflow here.

So we're going to look for another layer to provide access in a safe way. So we'll dig a little bit into automation. We're gonna build for ourselves and what we want that to look like and then we'll talk a little bit about how to take that Automation and make it nice and delicateable, right?

So looking at some key benefits, when we invest in automation. What do we want to get out of it? Right some of the main things especially in modern environments is sort of teaming that complexity, right?

We think about how many potential tools and runtimes and platforms and environments and all these combinations of things that we could possibly have to run our code and it's crazy. There's just all this stuff and as we learn about each component, we pick the right settings that we want to work for us. Then we need a way of sort of encapsulating that so that the next person to come along doesn't have to start over and say, oh well, I don't know if I want this or this or this we can just say, you know, what here's the piece that works and we have now some kind of inventory list or a piece of automation that says hit the button and you get the magic environment.

When we have all these different components though, they change all the time, right? Everyone's doing continuous delivery. So you're on the receiving end of continuously developed software from your vendors stuff changes all the time.

All these third party Services all the resources get changed your languages your runtimes. All those things are changing all the time, right? So we want to be able to get ahead of The updates that we need making sure that we can change what our platform looks like for security releases and all that kind of stuff.

It gets super hard to keep up with that if we're doing them all manually, right? But we also want to then have some kind of mechanism in case we need to freeze something say oh, you know, this next thing has a breaking change in it. We want to make sure we don't go to there.

So I want to be very cognizant of how much change we want to bring into our environment. When we need it, right? Unfortunately as we're doing some of these things manually it's really hard to not make mistakes.

Right? It's super easy to if you're copying and pasting something out of a Wiki page right to accidentally skip a line or not notice that along line has very long options and it just keeps scrolling off the box or whatever. You have to keep scrolling and scrolling scrolling to get it all and miss that last important option and then not know what happened or to copy and paste something into the wrong terminal or the wrong window and have things go completely wrong, right?

Totally happens. It's happened to everybody but we want to get to a place where we've got some Automation and controls in place that are going to help us avoid those things. Right and finally.

we want to reduce toil and we're gonna Thinking about toil is sort of sort of fancy sres speak for getting rid of the boring stuff that we already know how to do but it has to get done anyway and in SRE land, we're talking about things like security updates and provisioning all this things that happen all the time and you need more of them as your environment grows and for us in our example Cloud Ops, they know how to deploy stuff in the cloud for the developers and they just have to do it all the time. So we want to get to a place where we're taking less time doing those total tasks and gives us more time to do strategic tasks. So what gets in the way of this, right?

We have a whole bunch of different kinds of users in our organization. We've got our developers. We've got maybe some support Engineers.

Maybe we've got some database administrators or database reliability Engineers. However, you you organize those folks. They have tools they've built around their workflows that help them get their stuff done faster.

Our Cloud Ops team has a whole collection of things that they do, right? And so once in an organization's super large, you've got tools all over the place. Like there's just stuff hanging out right and some folks keep them on maybe there's a jump box somewhere or they're in this GitHub repository over here or whatever they are.

Right? Unfortunately. If I wanted to give Alice access to the tools that cloud Ops has built there's some things that get in the way right?

Number one. We've got calling knowledge Gap like Alice knows enough to be able to say I want to deploy my stuff in the cloud and maybe she knows which particular Cloud platform you're in this week, but probably not to know specifically what image security has blessed for today, right or how we put the private networks together or some of the other security settings, right? It's not that she can't know it.

It's just that she happens to not know it right now. Further than that, we might have a skills Gap. Maybe we have some folks who aren't super comfortable on the command line.

They're not really jazzed about getting in there on the CLI and typing in all the options and dealing with all that stuff and that's okay, too. They're also you know, lots of environments where it might take weeks or months to get super familiar with what your vendor provides and how to use it and how to set it up and and all that kind of stuff. And then finally we've got an access Gap we've talked about that already, right?

We're like not everybody is allowed to touch the thing and they're might be a historical reason for that. There might be a security compliance reason for that. Just depends on the organization, but it happens often enough where everybody who needs to consume that platform isn't necessarily allowed to deploy that platform.

So we have a bit of like stranded knowledge and Stranded tools and this Gap in the middle. So when we make Automation and when we put tools together for ourselves, there are some things we want to think about before we can get to the point where I can say to Alice. Hey, you know, what?

Here's this button hit the button when you need to stand you check environment, right? So some things to think about before I get to that next part, right? Is when I build my automation Solutions or that I am, you know searching maybe for a vendor to help me provide those I want to look for some characteristics.

They're gonna help me over the long life cycle of my development team and the product cycle and all those great things. I want something that's just gonna work today and I'm gonna outgrow, right? So if we look at these I borrowed this particular list from Lee Atkinson's book architecting for scale and if it looks like the same goals, you'd apply to software development.

You're on the right track, right? Because we're looking at things that are going to help us over time. Right?

So if I'm thinking about how I want my automation to live in my environment what I want it to do for me, I'm gonna make the investment of putting these resources together. I'm gonna get some benefit out of that right over enough of a period of time to make it worth it. So Looking at how we want our automation to sort of fit together, right?

We want to be able to test it whether that means doing test-driven development or just putting some unit tests together or whatever. That means like having a plan for this piece of automation whether it's a script or program or whatever. When I get out of it should be X Y and Z if at some point, it does not produce X Y or Z.

It should fail, right? Then we want to be flexible. I've got Alice over here.

But I also have Gary on the next team over and their environments are gonna look a little bit different because they have different dependencies, but I want to be able to make use of my Automation in ways that I can apply it to both. Alice's team and Gary's team. So we wanted to be flexible right?

Maybe that means taking in some options. Maybe it means environment variables. Maybe it's pulling things from Vault whatever that could be.

Right? So we want to sort of expand things out not hard code stuff then we want to Have good life cycle practice, right? We want it to be reviewable by someone else on my team.

Hey, I put this thing together. Let's let's walk through it or would you take a look at it? And that comes along with putting it under Version Control, right?

sh underscore do not use or something with a random date stamp on it like stuff just kind of builds up as you make changes to it. But we want a good workflow practice to avoid all of that to avoid confusion to make sure that the expertise that we're encapsulating in this little piece of tooling is exactly what we wanted to be when we need it to be there. Right?

So having good clean practices around that and that helps us get to the last part the repeatability and auditability of the things that we're doing especially for automation. Right if Alice is going to request a sanity check environment today. I want somewhere to be recorded that Alice requested this environment at this time.

It was approved via the finops request or whatever and then deployed and I want all of that to come out in some kind of audit, right so it needs to like keep a trail right so that we know these things are going on. And this helps us in other parts of our workflows, right? We're pager Duty.

So we talk a lot about incident response process and how that can be improved and automation is a big piece of improving that as well. Right? So not just with deploying things from my developers.

But what happens when Alice has an incident that she needs to start triaging and debugging in production, right? We have our responder there Alice the application developer. Her application and she knows some pieces of the environment the libraries and how they work and the runtimes and how they work but once she starts to bump up against the edge of her knowledge and the things that she has access to just a call in other people, right?

So maybe she needs to contact an infrastructure engineer just like I'm getting some kind of weird IO air in my log. I'm not sure what's going on there and maybe they can't figure it out. Either that they have to call somebody who knows more about networking right or looking at something else maybe in kubernetes and we're digging around in the expertise of all these people just to find sort of the contributing cause of what's going on.

And what Alice is seeing in her service right to keep asking other people for help and what these extra Engineers is really really doing. These are their first roll through all this just gathering information to show whether or not their particular set of applications could be the culprit here or is also just another symptom, right? So this example response focuses on incident response, but you know, it could also apply to other things that folks.

Show up with for, you know, getting Devin QA environments and way that doesn't release waste resources and also keeping things, you know separated so that we don't have to give our Engineers access to all the cloud accounts. So our goal might be Self-service automation right the ability to take the things that are experts build as our Automation and delegate that to the folks who aren't experts but know what they need right when we're talking about self-service automation. A lot of the solutions are focused on you know developers are good example of the end user right?

They're the folks that are going to benefit. From be able from being able to perform this sort of automation. They want to you know, keep their workflows working.

They want to keep developing and they don't want to hit these speed bumps or these little roadblocks when they have to request something from another team, right? So it's a good place to take a look at the workflow and See where the bottlenecks are see where those little roadblocks are so that you can smooth those over and keep the developers developing right? There's plenty of other folks who might have things that would be super helpful to them.

You might have folks in support that should be able to like run a little diagnostic or you might if you have a I got a special tier of level one responders or maybe knock type responders. They have some knowledge of the systems and some amount of sophisticated access to things they should be able to run some stuff too. Right so they might also be users that you keep in mind, right?

But they'll work in different ways. Maybe Alice and the other developers are a little bit more Savvy on the command line. So you can give them access to command line stuff.

We might have our knock Engineers who really want a web UI because they're looking at a million things. They've got blinking lights and flashing this and things are red and they've got enough distraction to try and figure things out on the command line and a button in a web U im a much better. We might have other Engineers that are working with us that want to make our automation part of their workflows.

Maybe they want a web hook, who knows right? But as we're working with, you know, all of these folks we want to give them access to what we've built in a way that makes sense for them in a way that Smooths everything out as much as possible. Right?

So we want to help them get their work done. And have us need to do less of their work for them, right? Hey, right so we want to provide results that make sense.

Whether that's a warning messages or error messages as well as the appropriate documentation for here's your environment. Here's how to access it. And then we want to provide a consistency of experience.

Whatever that looks like for those teams if they hit a button for one request the next time they need something they should be hitting a button for that request as well when it just help them help themselves. So over time we're turning our expertise into automation instead of having that expertise sort of stranded in the island of cloud Ops. We have bundled it all up into little pieces that we can delegate to Alice and to her colleagues so that when Alice wants a new environment she logs into our automation platform, whatever that looks like and she says hey, here's the job for sanity check environment application a and she hits the button and it goes through the job process.

Maybe there's a API check to make sure that hey Alice's team hasn't deployed too many things and has old environments running around or they haven't blown their budget for this week. If it's all approved, it just goes on creates the cloud environment for her reports back and says Tada. Here's your Cloud environment and somewhere.

It has recorded that Alice requested a cloud environment on this day at this time that job ended successfully, right? So there's lots of places where we have a chance to report back status. To check on the efficiency of all these components and they are can be super small.

Right like the little pieces that are doing that finops check maybe a two line shell script the pieces that do the cloud deploy might be something a little bit more extensive, but what we wanted to be flexible so that we can put lots of different things in it and we can start to think about what those lots of other things can be right if we're looking at just incident response a lot of that stuff. Is just information gathering, especially the stuff down in the green slice right performance checking information gathering. They're simple tasks that don't have a lot of steps and they don't change anything.

Right? So we've got impact on the vertical access and sophistication on the horizontal access. You can think about how things get more sophisticated and more impactful and maybe a little bit more dangerous.

So they're in red right things like adding and removing capacity or making firewall changes that He might not get to as part of our automation Journey right? We might be super comfortable automating things down there in the green which still allows Alice to respond to incidents and not necessarily. Have to Ping the infrastructure engineering team or the networking team for certain Diagnostics and other help, right so we can start to think about how we how much automation we want to provide across the systems.

And then we can think about how the automation might evolve over time. We sort of categorize these into a five-step Evolution, but there's not it's not necessarily complete Evolution a lot of the things that we're going to do. Don't go through all of these steps, but if we think about them, the first one is automation opportunities.

It's just a fancy way of saying things we have an automated yet, right they're out there, but they're still manual processes. And the next one sweet spot for a lot of this stuff right is human initiated automation. Alice knows at time x she needs a thing.

So she's going to initiate some Automation and that's a really big use case for a lot of these components right over time. We might have stuff that's a little bit more reactive to events in the environment. Right automation with oversight automation.

The feedback are both sort of Places where the automation runs and response to a trigger and maybe it needs a human approval. Maybe it just needs humans to look if something goes bad, but they are still out there sort of doing their own thing most of the time and then the last one the Monitor and evaluate is where the automation takes over completely right and humans don't really have to get involved at all and not a lot of our stuff will get to that point but there are certain things that probably should if you've got an annoying application that runs out of resources or has some kind of bug in it that you haven't gotten the time to fix it needs to be restarted on a regular basis. Like humans shouldn't be doing that right like some automation should be doing it.

It's on the host or it's in a centralized place or wherever it lives. We don't want to tell human beings he could restart that thing again, right? We took a look at some of our incidents over the span of a couple of years.

We found that like 20% of incidents that come into page or Duty are responded to and resolved in under five minutes by human responders. So there's a lot of stuff out there that probably falls into this category where the automation can just be triggered by the environment run on its own and that should just report back like create a metric. So instead of saying like Alice and Gary had to restart this application end times last week we get to say the application restarted self-end times last week and we can see how that compares the week prior.

Is it going up? Is it time to prioritize fixing it? Is it going down?

Oh, maybe it will take care of itself and it's something in the environment, right? Who knows we can start thinking about things that way as well. So there's a lot of things to sort of keep in mind as we're looking at what we want automation to do for us.

Are we just looking for? Getting out of some toil and giving those tasks to other users and that's great as an excellent place to to build a lot of time and put some work into that or we have things that we really just want to get humans out of the loop completely and there's definitely value there right where we're saving those resources and human time for other stuff. Right?

It's all about getting our time back and having more time to work on things that impact our users more directly right building more features into our application doing more strategic planning about our reliability all those great things. So if you'd like to learn more about how we think about this and some of the stuff that we have in our platform that supports it. Love to have you investigate some of our resources.

com. com. And then we've got a resources page for you, which is links to a bunch of documentation.

We have like a long long form white paper about automation Plus Code examples and videos and all kinds of opportunities there. So we'd love to hear more about what your automating and what kind of Journey you have and if you have any other questions Reach Out Thanks for coming to my talk.