Bob Wise – GitOps-Based Ops Modernization

I'm honored to introduce Bob Wise, General Manager of Kubernetes at one of our sponsors, Amazon Web Services. Thank you, Amazon, for sponsoring this Commit. Bob's session is titled "GitOps-Based Ops Modernization".

What is gitops and how is it changing ops even more than Dev Ops has? 0 release. He's also been a governing board member of the CNCF.

So if there's anyone who has great insights on where Dev Ops, Kubernetes, and software delivery in general is heading, it's Bob. Let's check it out. And don't forget that you can ask questions in the chat during the talk.

Hello, I'm Bob Wise, the general manager for Kubernetes at AWS, and I'm going to be talking today about a topic that I have a lot of personal passion for, but I think is one that is becoming increasingly important to our customers. And that is this topic of gitops. I've titled the presentation here "Evolving dev ops to gitops" ups because I view gitops as a proper subset of dev ops, a more opinionated way to think about dev ops approach.

But as always, I think it's important to think about the big "Why". Why do we care about the devops? Why do we care about gitops?

Why do customers care? And I think these are the reasons that I hear from customers all the time. Velocity, overall velocity of building systems, building software, delivering value is critical to competitiveness.

And this isn't just true in IT, this has become true across all industries. And we've also been seeing in a number of industries huge velocity gaps between the high performance orgs and the average performance org. Now, I wouldn't say big gaps between the high and the low.

That's certainly true, but even between the high and the average here. And then combined with that is a natural desire to be more efficient, constrain sprawl, use fewer better tools, have everybody use fewer, better things. And for teams this is an HR issue.

Teams need to modernize their approach because it helps employees be happier. Happy employees help recruit their friends, and a good work environment is good for everybody. So there's considerable pressure to evolve.

This isn't recent pressure. This has been going on for a long time. And this has, of course, been leading to a lot of thinking.

Some of it more recent like gitops, some of it has been in process a long, long time. I'm talking about the importance of how do you how do you make more small changes as opposed to fewer big changes. And this is, of course, the discussion that's been going on for years and years about waterfall versus incremental kinds of approaches.

Dev ops is strongly in favor of a small incremental change approach. Gitops has a kind of more opinionated way to think about the specific approach for how to do that. But overall here, what we're trying to make sure is that even if you start with the greatest team and the best intentions, if you're in a mode where you do lots of handcrafted things and it takes too long and you're not using enough automation, you go slow.

And then when you have a problem, the reaction is to add more time and add more process, which means the next thing you have to do gets even bigger, which increases the chance of a failure just based on size. Whereas if you can make progress towards a more high velocity approach with the smaller changes and it isn't just about small changes, it's about how do you make small changes and use automation, get the help from the robots to make that happen. So gitops as a way to potentially kind of help move from spiraling slower and slower to spiraling faster and faster.

But that said, fully automated deployments are really, really hard, so I don't want to minimize that because as important as it is to kind of leaping that chasm from, let's say, average velocity to high velocity is a big investment, it's a big change. And hopefully some of the things I say here today can help on that journey. So why is that so hard?

Complications. That's a big, complicated systems. And trying to coordinate all the changes between the two systems is really hard when you have systems that have lots of ordered steps, when that number of steps gets really large and gets really dependent on each other, things like a complicated.

Dependency tracking: there's a lot of the success of immutable style approaches and containers generally are based on improving the dependency tracking situation. And then you have packaging at various levels of the system. How do you deal with versions, different versions of different versions of let's say you have lots of micro services, each one of which is versioning differently.

You have even more complication there. And then what do you do when things go wrong? You have to have an approach for rollbacks when continuous, continuous when a particular deployment isn't working.

So. So these are all these are all the complications now. And where we've been moving to as an industry to approach these things has been to take some pretty bold steps.

A shift from imperative approaches to declarative approaches is certainly one of them. Moving from kind of automated mutation to mutability is another big one. So one of the approaches that that teams took to approach the automation was to have systems that kind of reach out into production and change them around a lot.

And one of the things that has worked really well is taking a more immutable approach, a container. You don't go and upgrade a container in production. You throw it away and you replace it with the new one, a new one that's been tested.

So immutability approaches have been really critical. And then on these larger, more distributed systems, you can't depend on a single state system has too many states. It's too complicated.

And we have to absorb and live with that complication in a good way. And the approach that has been working really well is to view reconciliation and convergence on an ongoing basis as a critical technique. So to be a little bit more specific, right, so manual processes, automation, this has been going on for a while.

This is at the dev ops level, really what's going on here? But in a gitops and containered world, things are a bit more prescriptive and there's some better opportunities here to improve. So I'm replacing package managers and bash scripts and so forth with containers.

Actually, depending on configuration is code having all of the configuration be in a source control system. Again, best practice in the developed world. I'll get to this in a bit.

Gitops has even even more opinionated view of that and then including a security posture in what you do here is is critical. One of the big I would say a realization for me here over the past number of years has been that focus on CI is really important, but lots of teams have also solved the CI problem. CI is well understood.

Lots of teams do it. It really helps the development teams go fast. It's a critical dev ops practice, but continuous deployment is actually quite hard.

And so as we move from a world where the dev teams are going really fast and the operations teams are maybe not at not going as fast, I think this does shift from focusing on CI to focusing on CD is really important. Ok, so. Now to the main topic here, Dev Ops is Dev Ops is transitioning to gitops for a lot of a lot of folks, especially in the container world.

OK, so first of all, I can't really say git ops without tip of the hat to the Weaveworks folks who invented this term. And now let's move on to kind of what what are the principles of get up so. Systems described entirely declaratively the desired state versioned and source control system approved changes can be automatically applied.

Note here that the shift here is a bit towards humans do their work up front and the robots do the work after that. And then the use of systems like Kubernetes that have great support for this kind of model, the built in support for building agents and watches and things that that work really well in a gitops world. So just to emphasize here, the declaration, versioning, automation, and then support from the underlying systems.

OK, so let's dig into each one of these a little bit more. Declarative is important because it's much easier to reason about it, scale it. It forces simplification and a disciplined way.

There's always things that you kind of want to think about imperatively, but ensuring that the teams are required to think through a declarative approach often helps simplify things. A really important point here. It's it's defined intent.

So this is the how do we how do we in this new world manage robots? How do we cross the human robot interface? And the the approach here that's been working is to define intent.

And like robots do the work. And then a notion of constant convergence of the system is really critical, especially in these large distributed systems, declarative approach helps make a clear convergence point that the robots are always working towards. Everything in source control.

So this is, I think, to dev ops practitioners, this would be noncontroversial, maybe even outside of that. This is a good software engineering practice. Really critical here, though.

You want a human readable source of truth, want the humans to be stating the intent. And one of the ways you manage the human interfaces here is this is a great place to have multiple eyes looking at a system. So when you want to have multiple humans looking at a thing, using the well understood kind of source control code review, kinds of mechanisms work really well.

And then in the in this kind of workflow, typically deployments are triggered by merging the PR. And then the robots get to work. We love our robots.

we still need to be nice to them. I would say that this is probably the biggest difference. If I wanted to poke at that, that difference between the really high performance org and the average drug, I would say this is it.

Most organizations are really, really worried about humans declaring a thing and then automation taking over and taking care of the rest of it. And this is where you can see opportunities for a spiral to slowness, where you have lots and lots of process and checks and other things that humans are doing in the middle of a process rather than doing that up front. At Amazon, we are terrified if our orgs are deploying software directly to production and not using automation.

So I think this is this is a place where we're we're practicing what we preach. Again, though, this is really hard. This is a big investment.

But the payoff that you get for making this investment is great. All right. So why Kubernetes?

Kubernetes Was really built with as as a robot? It's one way to one way to think of it is is is a robot for ops teams. And it supports has from the early days of declarative kind of approach.

The notions of convergence are built into the system. This is why the controllers then operator's pattern is so popular. Immutability, its container management system, it has support for things like deployments, which helps here, it has these kind of convergence properties around organic healing properties of "oh, a thing failed, I need to convert it back to the back to the desired state".

So a lot of the basic things that you need in order to start taking a gitops approach are there. So just to poke at this a little bit. I've talked about this a few times, just to make sure we have the have common context here.

So continuous convergence, drift management, you'll hear the drift word often and get up to conversations. So the idea here is you should have the desired state. And that's really that immutable thing that's checked it into your source control system.

You have your desired state. ". And then take some action to either correct it or it can be the case that because the robots are still not not that sophisticated, sometimes they still need human help.

So often the robots can fix a problem. Oh, a pod died. I'm supposed to have five pods here.

There's only four pods. Let me converge the state back to five pods, for example, in Kubernetes. The robots can do some of the work, but maybe it gets too complicated.

"Oh, I ran out of resources to get back to five pods again. So how is it that I .. " So a big piece of the story here is ensuring that it's clear in your systems where where the robots are doing the work and where they need to ask for human help.

OK, so digging even a little bit deeper here, if you want to think about this another way, I think this was again when I started first started trying to understand the implication of getups operationally, the notion of the source control system as a firewall between the CI world, the developer world, and gitops world, where the ops world, the deployment world is really important. So if you want to think about how do you fit the notions of CI and CD together, this is the way I would I would suggest you think about it. CI as a kind of build process where artifacts – By the way, one of the things that's missing here is from this diagram is an artifact repository, like a container registry, for example.

– But if you'll forgive the simplification, the main point is here that in the CI and dev world, you do a lot of things, they're they're committed into the source control system, the humans review it, and then the robots take over. So you get a single source of truth, all operations, all changes of the system are committed by pull request, and then you get into this checking the diffs and automatic convergence. This has another really positive benefit, especially from security and audit perspective, which is you have a clear, very clear kind of change control path.

Here's the humans that looked at the changes. Here's who approved it. Here's who merged it.

Here's the desired state of the system that has a lot of positive effects outside, even outside the operations team. I have to mention here as well, in the Kubernetes world, if you're if you're practicing gitops in the Kubernetes world, Flux important or extremely useful component here that does the git cluster synchronization or source control cluster synchronization. OK, so in summary, Gitops is a standardized workflow for configuration and deployment, updating and managing, well, broadly infrastructure infrastructures code.

But in this case, the more narrow view here is Kubernetes and all components. I'm going to say another word about the all components piece and then applications as well. OK, so one of the things that we announced last week at KubeCon is a system we call is ACK.

It's in developer preview. It's open source. Come check it out.

Try it. Give us your feedback. But the reason I mentioned in the previous slide, the all dependencies, is that to date it wasn't entirely clear if you're using a gitops approach with Kubernetes or EKS, for example, on us, how to think about application dependencies on services.

So we have a new thing here that we think is going to help our customers practice. Gitops example would be you want to define in our database instance and it could manifest. You can you can clearly do this kind of work with or without ACK.

We would certainly recommend if you're Kubernetes-centric and you like like to work in that way, that this would be some help. All right, so effective strategies. Where are we seeing customers actually put this to put this to use and what are they learning?

I think the main one has been focus on operations on the operations side. And if you're if you're in an organization that likes to best practices, you build it, you run it. This is more of a lifecycle and thing.

Maybe within the team. If you're a operation that uses a SREs or perhaps has development and operations in separate teams, you can you can think of the focus here as being on the operations team as opposed to the operations practice, either way works. And again, the focus here is on operations agility as opposed to developer agility.

You can, of course, use container orchestration to help realize CI and CD. And I would also say that this should not be viewed like most things. These kinds of transformations are really hard.

As I said earlier, continuous deployment is really difficult and it's important to take an incremental approach to this as it is anything else, and you can incrementally evolve the entire organization, not just not just new projects. So, for example, we see customers have had a lot of success in taking an application that perhaps in the longer run the org wants to reengineer into a micro services style of architecture. First, containerize that application, get continuous deployment working for the monolith, build out the organization and practices incrementally in order to achieve that, and then they can start to incrementally evolve that or maybe not.

Some applications in that state can use a lot of that kind of self healing properties and immutable approaches to understand and manage changes, but don't really need the additional investment in micro services kind of approach. But I would say treating this kind of operations modernization as a critical prelude to any kind of micro services architecture is critical. So my recommendation here would be to make sure that you can do continuous deployment on your monoliths before you start before you start re architecting those into micro services.

And beyond the topic of this talk, continuous observability is as important as continuous deployment. OK, so other approaches that we've seen that seem sensible also in a kind of incremental way, and this is a bit more common for four companies that have split their their platform or ops teams separate from the applications teams. And you can indeed use a gitops approach at both the application level and at the platform level.

And so we what we see sometimes is the application teams absorb this kind of approach first. But what I would say seems to work a little bit a little better and is pretty common is where the platform team first tackles a gitops-based approach for their kind of internal platform and infrastructure management, and then over time that moves upwards into the application teams. So there we have it.

Thank you very much for your time.