Bob Walker, Octopus Deploy | DevOps Experience 2022

Thank you everyone for joining me today. My name is Bob Walker. I'm a technical director over at octopus deploy.

And today I like to talk to you about setting realistic devops goals. Just a little bit about me before we go ahead and get started. net developer way back in the early days of Net10 and 1X.

net web applications. Now I've been very fortunate to work for a variety of companies and I've encountered a variety of deployment philosophies. Some are very manual and others are very automated but I would say Far and Away the worst was my 2 am Saturday deployments the big reason why is because they were very manual but the unfortunate thing was is that was actually written into the contract.

That was the only time we could do an outage window. So as you can imagine we weren't doing our deployments every week we would do our deployments once a quarter maybe once every couple months if we're being a little daring or anything along those lines. Ever since then, I've been really into automated deployments.

Now. I've made a number of different mistakes. I've tried to automate the wrong thing, but I would say probably right around that 2 am Saturday deployment.

I would when I had to do those, I really started getting into that. Now my specialty is database deployments rollbacks, but I love talking about any sort of automation devops or anything along those lines, but those two topics specifically really get me excited. As well as you know, setting realistic devops goals, which is exactly what we're going to be talking about today.

So first up we're going to talk about an unrealistic goal scenario and why it's unrealistic specifically. These are going to be something that I faced as a lead developer a few years ago. Thinking about some goal considerations and what you need to think about before you start picking off goals and start doing that type of work.

And then finally how to set realistic goals. Now I'm going to pick on a surprising unrealistic goal because most folks probably think no we need to deploy to production every day. And I was in that camp as well.

I wanted to deploy to production every day in the middle of the day with no downtime. I think that made a lot of sense to me and I was going home about it. I wanted to do it put my blinders on we're gonna do this.

Now a little bit of application context because I think it's really important to understand the application that I was working on. First up it was an internally developed loan origination system used by about two to 3000 people. Now this loan origination system was a designed to help farmers get loans.

And if you're familiar with farming in the United States, you know, it's a very expensive Enterprise. So we'd often see loans ranging from $50,000 all the way up to 10 million dollars because farming in the United States is expensive. I was working ostensibly for a bank slash Credit Union and we worked Bankers hours.

We had retail offices that we had to worry about in the Eastern Time Zone as well as in the Mountain Time Zone. I'm located in the central time zone. So it was seven am to 6 PM was when the offices were open.

So naturally our deployment window was 7pm Monday through Thursday and on the weekends, we didn't really do Friday deployments. Mostly because it was due to Resource constraints getting people to actually do them as opposed to technology constraints or anything like that. speaking of constraints we had a few application constraints we had to worry about First up as you can imagine giving anyone 10 million dollars.

You're probably going to want to check a number of different things. So we had very complex business rules that we had to worry about. Not top of that anytime.

We had a new features. Oftentimes that would coincide with other application updates and that's because we were relying on about 15 back in services. And these Services were something you would expect such as we're going to check the credit score and if someone has a you know, 300 credit rating, we're probably not going to give him 10 million dollars.

But at the same time there's other unexpected things like, you know, do we have financials as well as the usual stuff such as logging in our Authentication? now oftentimes if we ever introduced any new feature We might have to wait for an update from those other applications those backend services. but I had just gone through a massive effort to get from two plus hour deployments that we're very manual at the time and we are down to 15 minutes.

And I thought like we're so close to zero minutes. We're so close because we have 15 down to 15 minutes. And we did that.

By going through a massive overhaul the entire ci/cd pipeline. And I areas we improved other areas. We completely rewrote the whole thing.

We adopt a new tooling. For example, we migrated off of Team Foundation Version Control on to get. And that allowed us to introduce new branching policies be able to create feature branches or do pull requests.

And as well as get automation so we can move checks to be earlier or fail faster. We also got consistency across all of our environments which was fantastic for us because before you know, Devon test was very different than what we were seeing in staging production. And now we are getting this consistency across everything.

Because of that we reduced a lot of our dumb mistakes that we were seen. And some of those dumb mistakes were we'd forget a schema update to our database or forget to update a configuration file. And by doing that we were building trust not only in ourselves.

We know that yes, we can successfully deploy something. And not face a whole bunch of emergency fixes the next day or over the next week, but the business also started trusting us as well. And we ended up with a really good pipeline of features and fixes.

We had the steady stream. Every couple weeks we would do these quality life improvements. So this would include bug fixes, maybe a few UI tweaks.

Small features nothing super major and maybe a few security enhancements. Now every month or so we would have a much bigger outage. And that's when we had introduced those new features or any sort of new integration or anything that required a little bit more work.

So when I say it's a 15 minute deployment what that means. It's an average. Sometimes it was five minutes.

Sometimes it was 30 minutes. Sometimes it was 12 minutes of the times. It was 16 minutes.

It's over average over time was about 15 minutes. Now because of that we had the ability to do critical fixes relatively quickly with low risk, if necessary, we could deploy in the middle of the day. We would take that five minute outage window.

But we would get the code out there and most people probably wouldn't even notice especially since we probably do it over lunch time. But as we researched. This more and more we started coming to the realization that we're going to make some pretty fundamental changes with how we deploy and how we develop.

Because we might have to introduce certain deployment patterns that we haven't introduced ever before because you know, we've always done these outage Windows what we're reducing was the outage window overall, but now we sort of thinking about can we do blue green or Canary style deployments or having some sort of staging style deployment and what that really boils down to be was we wanted to run two versions of the application source code pointing to the same database. That was because our database was rather large and we were leveraging a relational database. So we started looking at you know, what kind of design patterns are out there.

And this would also fundamentally change how we introduced change into our code base. We would we were looking at the expand and contract model, which if you're not familiar with is making small incremental changes to overall introduce a very complex change. For example, I want to move a column from table a to table B.

The first deployment I might update the code and add the column to table B and that code update will write to both table a and table b as well as read from table a and table B. That I might have another deployment where I introduced my migration path to get the final data pushed over there. and then I'd have another deployment where I would remove all code referencing table a And then finally, I would delete the column out of table a so multiple deployments.

So that would change completely how we did any sort of our how we actually developed code. The other thing we're looking at was why don't we leverage feature Flags? Which we did at times but what we were finding is oftentimes we'd have a lot more if then statements.

As well as we'd have to think about. Okay, how does the UI react? How does the code react when the feature fact is both?

Disabled as well as enabled. So we'd also have to worry about a whole new suite of Automation and testing. So what we're facing was basically the same amount of effort and resources.

To go from 15 minutes to zero downtime as it took to go from 120 Minutes down to 15 minutes. The one thing I wasn't considering and outside of all of these additional coaches and how we're going to change our deployment patterns. was our user base because oftentimes they're the reason our application exists.

And really it's the abilities that excuse me, the person's a capacity to absorb changes. So if you're adding a new feature, especially if it's an internally developed application, there's going to be a lot of training because it could be extremely complex. Now get into what that training might look like in just a second.

But if you're making a sort of UI changes, you're also moving cheese, and if you're not familiar with that obsessively what it means is how I did something yesterday is not how I'm going to do something today. This also going to be this change of expected behaviors oftentimes if you fix bugs Sometimes some users expect. That behavior in the buggy Behavior to be actually how the application should work.

So users might develop workflows or expectations and go, you know, this is how we expected to work. Meanwhile the other users. Are very vocal say no.

This is a bug we think this is a bug and as as developers. Well, yeah, absolutely. It's a book.

Let's fix it. So we solve one problem for a big group of people but introduce a new problem to a smaller group of people. It's also training.

So let's talk about training. It could be very intense. So you might have a lot of workshops with a massive amount of people.

It might have to write up some guides as well as make any appropriate announce announcements. And this is everything that we had to face for this internally developed application. the other thing is to think about is that every new feature you had will have bugs and this is an unfortunate truth of software development.

No matter how much automation you include there's always going to be some use case you didn't think about Or some Edge case that you never thought would ever happen happen and you will have to have fix that bug. So what you end up having is this kind of healthy. tension between stability and new features And oftentimes users they prefer that stability especially in the middle of the day when they're trying to get work done or at towards the end of the month if they're trying to close a number of different deals.

Oh. The question that I was asked was what problem are you trying to solve? And clearly our users didn't want this.

And there was a pretty big investment in time that we have to worry about. And unfortunately, my answer was I just don't want to deploy at 7pm anymore. I kind of wanted to deploy in the middle of the workday and make my life so much better and make the developers so much easier.

It lives so much better as well as the operations folks. And that really didn't fly. So let's talk through some goal.

Considerations. Now there's two big buckets and we've kind of been talking about them which is going to be both the user impact and the technical impact. The user impact is going to be dependent upon the application that you're developing.

What kind of traffic are you do you get per day? What is the user base? Now from a technical point of view.

You have to worry about things like your regulations and company policies. As well as your architecture. And the existing tooling and I kind of want to divide dive into each of these six different items just so we have a better understanding of what I'm talking about.

Now in terms of application types, it's important to note that not all applications are created equal. I was working on an internal application but there are public facing applications. And so there are things like e-commerce websites or home pages.

And they're very different from one another in terms of complexity and expected traffic. And on top of that you also have to worry about things like where I like to call Rambo services and the reason why I call them Rambo Services is because they can never fail. They always have to succeed and these are things like You're logging service or your authentication Service or the configuration service.

These are services that all of the other applications are built on top of You might have an internal application with you know, your own custom business rules that you have to worry about. Or it might be a business to business or SAS application where you're providing software to another company or it's cousin where it's more isolated and you might be hosting the software on the company's servers. Now in terms of daily traffic that also has a pretty big impact on any sort of goal.

Considerations. Because if it's a public facing website, you might have a situation where you get millions of visitors all day every day 365 days a year. Or it might be a little bit more Regional and you might have 18 hours a day, but it's typically business days Monday through Friday, excluding any holidays.

If it's business to business if it's isolated, you might only get a thousand visitors during normal working hours for that particular region, or if it's shared infrastructure. You might have a situation we get hundreds upon hundreds of thousands of visitors. For you know 20 hours a day roughly every day except for holidays.

On top of that. You also have to worry about critical events. and what I mean by that it could be a multi-day or multi-week situation where it's like the holiday season, so if you work say at a retail store During the holiday season deployments.

Could have a material impact on the company's ability to make money. If you can't sell something if the e-commerce website is down that's going to be a pretty big deal. Or it could be a single day event like a Super Bowl Sunday.

And if you are a company that sells pizzas or makes pizzas, you're gonna want to make sure that that online ordering system is up and running because that's a huge day in terms of profits. If you're a business to business provider, if you're introducing a brand new customer, you're going to want to make sure that they have a really great experience. Especially if it's isolated.

It's gonna you're gonna want to make sure there's no other deployments you've tested it. And it's as rock solid as possible because that's the first experience the users ever going to face. speaking of users How what you designed for and who you are you targeting is gonna impact their expected knowledge.

If something is designed for kids, there's a lot of different considerations, especially if it's more of an educational thing, so you might have to worry about, you know, does this change per semester per quarter? Is it per school year? What can you change?

What can't you change because you're kind of holding to the school year. Now if it's designed for the general public if you've ever had a chance to work with the general public you're going to know that there's a wide Gulf between someone who is an expert and someone who is a novice. But oftentimes you have to design for the novice because you want to make sure that everyone knows how to use your application.

So it might be a case of introducing very small subtle changes or feature that you highlight to a number of people. if there's an internal application, there's kind of this expected knowledge that's going to be there and you're going to be using the company's nomenclature or there's going to be a lot of training to get a person up to speed on how an internal application works. If it's a business to business application, that's kind of a mix between general public and internal application because there's a common exceptic expected use case for those business applications.

So it might be a mix of complex business rules, but it needs to be designed. So a variety of different user roles can use it, you know, for example at octopus deploy. We implemented Salesforce now the users who use it range from our account Executives or use it all day every day to manage.

You know, it's more of a CRM system. Versus say our product managers and they're looking at kind of Trends with that. They're noticing within Salesforce or what are they seen from their customers or like what's being reported from our customer base?

Now moving on to the technical impact you have to start thinking about regulations and company policies. So these are the things that might be even out of your control that you have no chance of changing whatsoever. Almost every industry that I've worked and we have some sort of Law and some sort of Regulation that we have to think about.

Generally it boils down to these two big rules, which is the person who made the change can't approve it or the person who made the change can't deploy it. There's gonna be other rules and regulations. You have to consider.

Those aren't the only two ones. On top of that. You also have to worry about contracts as I stated earlier.

I could only deploy it to am Saturday because that was written into the contract. That was the accepted outage window. Or you might have a situation where the customer might sign off must give me must sign off on any sort of change before it goes out to their isolated infrastructure or it goes out to their version of the code or something along those lines.

Now for company policies, you're gonna have to worry about how are we going to enforce that? So you might have to work with change approval boards to ensure that the business owner signs off. Or an appropriate amount of testing has been done.

So test plans must be submitted. now how are you achieve or are compliant with those regulations and company policies that might change with your goal and that's acceptable but Just you need to be aware of those before you start going gung-ho and start trying to change a whole bunch of things. Now the application architecture also has a pretty big impact on whether or not the devops goals are realistic.

And I put up there not because it's a monolith. and that's because you know, I've worked with some monoliths that are very You know, they're very well put together and the deployment process is actually surprisingly smooth. What I'm really referring to is what kind of coupling do you have with other dependencies?

In the loanervation system that I worked on there were certain cases where it's I try to make the coupling as Loose as possible. So if the service wasn't up my application didn't crash or anything along those lines, but if they added something new I needed to know I might need to know about that and be able to change my application to respond to a different response on top of that. What kind of software or tooling is required to just run that application.

What do you need to install on it? Is there a license you have to worry about? This also those design patterns when I say design patterns.

I'm referring to things like your repository pattern your factory pattern your strategy pattern command pattern or anything along those lines, but ostensibly what it means is does your application have the ability to support automated testing? And can it incorporate additional changes when necessary because even though you want to make all of these changes if you want to cut down on manual work. You need to have some sort of ability to verify that that work is successful and manually verifying is just going to slow everything down.

This also the code and database interactions the story. I always like to use was one of the code bases I worked on. The order of columns as it came back from the database.

Was important because people had gone in and referenced the index instead of the column name. So if you introduced a new column and you put it right in the middle say the select statement the application would actually crash. um So as you can imagine that was very brittle.

The other thing to think about is what tooling is already available that you can leverage to help you accomplish those goals. So for example Source control that seems to be the foundation upon which all of the devops tooling is kind of based on if you're using something that's extremely old and unsupported such as visual Source safe a lot of the goals that you want to accomplish. Can't be done until you change your Source control.

In addition what kind of build servers out there can it support a number of your goals can that doesn't have the ability to do that? And are you using a build server as a deployment server, or do you have a separate deployment server that can help achieve a lot of those goals. Now on top of that.

The other thing to think about is your database is the schema undersource control because having automation just in your Source control. This can be your source code. Doesn't really matter if you have all this Automation in one area and you have one other component, that is everything is very manual such as the database so you have to worry about the database automation as well.

So what tooling is available to you? So does it make sense to do it? Doesn't make sense to those goals.

I'm gonna go back and I'm gonna pick on the Zero downtime deployments in the middle of the day. And there are cases in which yes, absolutely 100% It makes total sense to do it. If you're a public facing application with millions of visitors all day every day, you're gonna have to be able to solve for that.

There's no other way around it because you need to be able to deploy and have a really great experience for your users. If your business to business application, for example, and the only time you can deploy is on the major, excuse me, the only time there's no traffic is on the major holidays. Then yes, you're going to absolutely have to worry about something like that because having 15 possible days to do a deployment on these major holidays.

No one wants to do that. It's gonna be really hard to get any sort of changes out there. Those Rambo services that can never fail Yeah a hundred percent.

Absolutely. You need to have a mechanism to solve for that. Even if they're internal services.

They are the foundation on which every other application is built. Now when it comes to know. If you have a business to business application and you're deploying to say customers infrastructure or a copy of you know, your hosting the infrastructure, but it's isolated.

Or require sign off before new version probably doesn't make a whole lot of sense. It might make more sense to wait until they're not used in the instance maybe during their their off hours to do the deployment or during some sort of maintenance window. Or if you have an internal application of business to business application, it requires a lot of training because of all the complex features.

It might not make sense to do it in the middle of the day when someone's trying to work. If there's a very clear and reasonable after hours outage window, you know, for example my case 7 PM that was pretty reasonable or 6 AM 6:30 am. It's going to be really hard to justify especially if it's almost every day.

When you have that outage window. If you have internal applications deployed to isolated infrastructure what I mean by that is if you are saying to retail locations point of sale machines. Probably don't want to deploy in the middle of the day at that retail location because you might take down the point of sale machine and prevent the ability for a you know, for that retail location actually make money which be a bad thing overall.

Or if you have no automated testing capabilities, it's going to be really difficult. to deploy in the middle of the day and verify everything is working before switching over. So let's talk about setting realistic goals then.

So what are some of the things you need to think about now? We have those considerations, but let's talk about how to set those realistic goals. So the first thing is ensure you have a valid business reason and that was the thing that I failed at which I didn't really have a really great business reason as to why I wanted to do middle of the data appointments other than I wanted to make some people's lives just a little bit easier.

The other thing is if you're not a hundred percent sure discuss it with others. Talk it over with your users and customers make sure that that you know, what kind of impact does you know your solution have on them. Will it have a big impact?

If you're unsure, you can also maybe talk to the support or your customer facing staff to get more input. But also talk in discuss it internally to determine even if it is feasible or realistic or we have the capacity to handle something like that. On top of that determine your priorities.

Don't try to do something where you hey, we're gonna do blue green deployments, but you have all these manual processes in place. You don't have any sort of automation. You're still using poor visual Source safe.

It doesn't make a lot of sense to start off with, you know, the absolute extreme. Have a very clear realistic priorities that make a lot of sense. and again discuss it with others to determine what those priorities should be because I could probably give you a number of my own internal priorities of what I think is important.

But again, it's going to be very different depending on your situation. Set those done requirements. When should you walk away whenever you reached an appropriate stopping point?

And then reevaluate those done requirements periodically, so imagine a situation where you said, you know, we want to get our automate. We want to automate our deployments to go from two hours down to 15 minutes. That's going to be our goal.

And you do a whole bunch of work and introduce a lot of automation you change a number of different things and your ci/cd pipeline. But the best that you can do is 20 minutes or 25 minutes. That's really good.

That's a huge win. And if you realize that the next to get from the 25 minutes down to 15 minutes isn't require a massive amount of effort. It's going to require a complete change and how you do things that it might be time to walk away and maybe work on something else.

That's a bit more important that has a higher priority at that point. So let's talk through a couple of other examples because I've kind of picked on the deployed in the middle of the day a little bit too much. So a good business reason might be hey, we want to be able to deploy bug fixes more frequently.

but the problem is everyone works on the main branch because they were using a non-distributed source control system such as git and creating branches is very resource intensive that build service don't support that any number of different things and we often have half finished work intermingled with finished work and difficult to kind of come up with a good stopping point saying, yes, we have reached a stable State and we're ready to go. This is the solution might be switched to a distributed Source control system such as git now. The user impact is going to be not they they really don't care how you store your Source control, but the technical impact is going to be quite big because you have to worry about new tooling.

What kind of training do you have to go through? Do you have to introduce any new build server changes? What kind of other policies do you need to have?

Another example of a realistic goal is maybe you want to prevent users from access in the application during the deployment window. Because during that deployment window what you're seeing is that deployments are causing errors and Bug reports and all kinds of other problems on top of that. And so maybe a solution is instead of blue-green style deployments maybe redirect all users to a maintenance landing page during that outage window or that deployment window.

Now the user impact is the maintenance are outage windows or deployment Windows they occur off hours. So it should be middle minimal impacts to them. In fact, it's going to be a benefit because they're not going to walk in on a deployment.

And start causing all kinds of errors or issuing bug report Sandy. This isn't working. This isn't working and you go.

Yeah. I know. I haven't finished my work yet.

So it might be it could probably gonna be a benefit to them as well. now for my technical impact you can have to worry about eating to create landing pages or have a process to update your load balancers to redirect all traffic to those landing pages and but still have the capability for internal users to access the system to verify it. But it is still a very realistic goal with a minimal amount of user impact and Technical impact, but it has a big benefit to them.

So that is it for today? Thank you very much for watching. Have a great rest of your day.