Shea Stewart – Google Anthos Cluster Change Control with GitOps and GitLab CI

Next up I'm happy to present Shea Stewart, Partner at Arctiq Intelligent Architecture. Shea's talk is titled "Google Anthos Cluster Change Control with GitOps and GitLab CI". T.

, but it's got a bad rap. Change management isn't inherently bad but the traditional ways of doing it caused inefficiencies. Shea discusses these challenges and how they can be overcome.

He also demos the solution in action. This is great stuff. Let's dive in Hi there and welcome to GitLab.

This is my talk on "Google Anthos Cluster Change Control with GitOps and GitLab CI". I'm Shea Stewart and I'm from Arctiq Inc. If I'm not behind the keyboard I'd really rather be out in the woods as you can see from my profile photo right there.

Now in my history I've grown up in the infrastructure side of the world and so when we started Arctiq what we wanted to do is move closer to the application. But my heart and my experience was on infrastructure and infrastructure change. So that's what we're gonna talk about here today.

Now a little bit about Arctiq. Arctiq is a Canadian based dev ops consulting firm. We are fully services-led but we operate in a bit of a different way.

We practice an organizational structure called holacracy. Holacracy, from what I understand is actually quite close to what GitLab practices as well, is a an inclusive, distributed, authority, and accountability model. And for those of us who live in the dev ops world it really relates to us as a continuously improving organization.

Arctiq is also a service delivery partner for a few major organizations, namely Red Hat, Google, and Hashicorp. So if they are performing professional services for certain customers it can and often is subcontracted to our team. They are also a Google Anthos design partner and have actually delivered over 30 Google Anthos.

platforms across a variety of verticals in both Canada, United States, and Europe. This is everything from major sports franchises to banks to major telecommunications organizations and so really what Arctiq brings to the table is the experience of merging in this new technology with enterprise process and that's what brings us here today at GitLab Commit. So we're going to take a look at the agenda a little bit and what I want to do is take a look at infrastructure change management as it's previously been done in the processes around that and how that can actually change with new adoptions from tooling and processes with gitops.

And we're hearing about gitops all over in the industry now and what I want to do is show you how it can be put into practice. Now what we're going to do is we're going to marry this conversation with a couple of products and solutions that I really. We're gonna talk about Google's Anthos platform, which is the hybrid cloud Kubernetes distribution, but more specifically we're talking about configuration management and how that piece of the Anthos kit helps us in our gitops change control.

We're also going to marry that with GitLab CI so that we can safely apply these changes to our environments rather than merging or changes directly into production or as we used to do in the OPs world, logging on and changing it right in production which is a thing we shouldn't do and so let's get started. T. organization.

T. organization provides to the business and around these services. We need certain things to be provided such as stability they need to be reliable they've got to be predictable we can't deliver a service to our end users and have the business function if none of these things are in play.

But aside from stability and reliability we also have to be able to change rapidly because the business is consistently changing. T. organization needs to respond in that way to enable new services or we also have to be responsive to the external factors outside of our business such as a zero security threats or competitive landscape changes.

And in this case we need to respond to the changing industry which means we need to also be able to rapidly push changes through our organization. This is really what change management is all about, how to do this in a safe and de-risked way. So while most change management processes are going to be more complex than this, those of us that grew up in in traditional infrastructure change management would recognize a flow somewhat similar to this.

All right I'm going to request a change that's going to be built and I'm going to do that usually by opening up a Word doc I'm going to email that Word doc to one person or maybe I'm going to email it to 50 people I'm not entirely sure. We're going to have a conversation around this and we're all going to evaluate the risk of the change and this is a paper-based evaluation and this process can often take days. Then, once we're done with that evaluation and we feel like we're comfortable, we're going to push that change into somebody who has authority to approve the change.

Again this is quite often an email or paper based approval process. Now all of this is speculative based off of how we think the change will actually be effected in production because we haven't tested anything yet. Then we're going to coordinate the change the coordination of the changes where it gets really tricky because we could actually spend weeks trying to coordinate schedules or getting the change inside of our actual review process across a different team or possibly into our blackout windows or around those windows that we have.

m. on a Sunday where this will be implemented. Meanwhile I as the requester probably don't work at that time of the day.

So there's a bit of a misalignment there with when the change is getting implemented and the person who wanted to change finally once we've got the human that's actually input the manual change and we're weeks down the road we can never review we can close off the actual change request itself. Now of course if there were a few problems that shook out of this but most of it worked we would call the chain successful and then we could generate new change requests that would go back into the queue and the process to fix any of the things that we broke. So that's a pretty traditional workflow and for those of us who have experienced it we understand what it can take to wait weeks or months for a change to actually be successful if it hasn't gotten through all stages of that process.

So how does traditional infrastructure change management really help the organization? Well it does a few things for us. I mean it is the process that actually allows changes to be made to services in the organization.

It's it is the process. We may need to refine it a little bit but it helps us affect service change. It also reduces risk because we've gotten a lot of eyes and a lot of input on the type of change that we want.

And it really maintains the stability of the service for end user. So we're always cautiously making sure that any change isn't going to affect them. And from the communication standpoint the change management process is actually allowing us to make sure the enterprise and our users are aware of this change as well.

So these are all really good things as it as it applies to change management for infrastructure. The problem with the process is that it's quite human, human prone, human involved, I guess I would say. And there's a lot of fragmentation on the discussion of the change in how effective it's going to be and whether or not it's actually you know we have a source of information running and production and make sure we're we're applying that change to and relating it against an and it can be quite error prone and slow and time consuming.

So this really doesn't allow us to often react to the needs of the business in terms of speed or agility. And it also lets us potentially in a more risky situation if we're blocked for months by a complex process that won't allow us to release things like security patches. These are all sorts of the challenges that the change management process can introduce because it's so human based.

And so you know paper based when we're using things like Word documents and emails and things like that. And finally one thing it's really not solving for from a stability standpoint is it's quite often difficult to reverse a change. I've often seen in my career that teams would rather push through the change no matter what type of damage it caused.

Go ahead and create additional change request to fix those later on but to push it through rather than try to roll it back because rolling back is often so complex and difficult. It's it's really hard to test and we don't know what the results are going to be anyways. Too often tend to push forward.

So these things really aren't helping our goals of ensuring that the services are sort of changing rapidly and responsive and in stable. So how do we fix this. Well luckily we've got development processes that we can lean on and tools that we can apply as we move to a faster more sort of agile operational model.

So if we take a look at teams who are developing code through agile processes there's a few things we can we can leverage here. One of them is really ensuring that all of the documentation about the change is stored with the code that's actually going to be making the change. That code is the thing that's getting implemented.

So so it's no longer a human applying the code the code itself is the change. This actually increases the visibility and transparency to all specialists who want to see what actually will happen rather than watching a few command lines that are going to be you know input into a terminal at 3 am on some Sunday morning. We can invite a bunch of people to collaborate on this change and within the same system they can communicate and make changes to that to the proposal in real time which also allows us to instead of ping pong Word documents back and forth they're actually updating the code in the change process itself.

And then once we're happy with that change we can actually kick off some automation tests to make sure the thing is going to work. And so the idea here is remove the human mental review process or or spend less time focusing on that and more time on automated systems actually validating the quality of the change and writing test cases to make sure it's doing what it should do. All of this can then result in the code review process as providing an approval stream and everybody can see the state of the change once it's been approved by the appropriate team.

That's when we shift over to the final sort of process which is the process of continuous deployment. Once that approval has happened the code is committed to the right branch if you will and automatically applied to a destination system. This system that should be running that code.

We can stage this application if we wish from lower environments to higher environments meaning that we can ensure through our approval process that the development environment actually only gets the code first before we approve the same change in a production environment. Thus did risking the change even further. And it's also really helpful through this whole process is that we have really inherently built in rollback process.

So instead of trying to force the change through we can revert to the previously known good code because it had previously run on the system. So all of this allows us to move a lot faster and with a lot more stability and quality. So when we talk about gitops a little bit that's what we're talking about.

We're talking about using development processes and tools and applying them to operational tasks. And so this is the revised workflow. If we take a look: all of this happening here on the screen that you see is is less humans and more code and more automation.

So as a requester of a change I'm going to go ahead and create a branch which is going to let me work on my own little safe space but I'm happy with my code. I'm going to commit it I'm going to run some level of automation testing so I don't have to bother other people to validate whether the code is actually good or not. Once my testing has come back and it's valid, I can actually open a merge request and I can ask for other teams for their input as well.

But they can know that as long as the tests are good that the code is actually going to do what it's supposed to do as well. So this actually helps us get higher quality on our code reviews. With all of this in place in the same repository and in the same tool we can then see the approval flow and of course there's a full audit trail of every change everybody's made here and who's approved it.

Once the approval is taken place which again you know a human in this cases is clicking the button we're actually going to move to apply the changes and all of that is fully automated and then we move to reviewing the apply changes which is a combination of automation and humans. But as you can see from the bottom of this chart all of this is down into minutes to hours instead of days to weeks. And this other switch that we have is at operational changes are often now able to be applied during the day so that the person that requested the change is also still working when the change gets applied.

Considering that person would know the most about the change that they've tried to make. So when we talk about the tools in this flow we're going to go back to a few tenants here and this is the point of this whole presentation. We've got GitLab doing all of the git and repository management system so it's holding all of the code all of the documentation and as responsible as the single source of truth for what is running on the systems.

Then we've got GitLab CI and CD and we're gonna use the CI piece to run our test and to do automation testing against our code to do lending to make sure it's going to be accepted by the system. We're on alert on particular failures and we're going to provide status of the health of that of the proposed change. Then we're going to use the continuous delivery components to actually make sure that anything that has passed and been approved is ready for release in the right branch.

And then we're gonna move to continuous deployment and in this particular case in the Anthos solution we're going to use Anthos configuration management as our continuous deployment tool. What this means is that it's got an operator that will make sure that the code is that is ready and approved is always running on the system and if there are errors it will help us to revert back and forth between the known good state and the problem state if we wish to debug any further. Again all of these these automation tools are going to help us and this visibility is going to help us ensure that our change is more stable and predictable.

So when we take a quick look at and those configuration management again as I described before we've got sort of our dev ops engineer or SRE or system admin or infrastructure software developer or whatever you want to call them they're going to offer the change they're going to push that change over to GitLab and that's the single source of truth. So it's going to hold all of our configs, it's going to hold policy like like security configurations and policies, and a number of other things that we can have up there like other Google resources and things like that. Now once we've approved that change really our cluster is running an operator here that has a couple of components one of them is running config sync to make sure we're always synchronizing what's inside of GitLab and then two, the policy controller engine that's going to validate that whatever we've tried to create is actually within our policy configurations that we've set out.

So our system is compliant. And that's the Anthos configuration management piece. What I want to quickly show you here is how simple and powerful this can be.

So what I'm going to do is I'm going to try to I'm going to show you the Anthos' dashboard quickly in here you can see underneath configuration management I've got two clusters registered. One of them is my GitLab Commit Dev cluster and one of them is my production cluster. What I'm going to do is I'm gonna jump into the repository that my dev cluster is synchronizing to and I'm going to do something that I shouldn't do.

I'm going to make a change and I'm going to push it to my main branch without any approvals or any configuration or change management. Now I'm just gonna rename this particular namespace and I'm going to show you first. These are all the name spaces in my dev cluster right now.

So you can see that I've got my monitor app namespace and give it a new name and I got to commit that change. Now the way that Anthos configuration management works is it actually isn't going to like this particular syntax and it's actually going to break that system if we go ahead and take a look at our status over here in a second we're going to see that this thing is errored. So I've done the thing I shouldn't do and made a change to production without any change management.

And guess what. Now I've got a cluster that's in an errored state and I can't do anything about it. Well I can.

And that's what I'm going to do here. So let's talk about that quick reversion but I'm going to do is go back to that commit and I'm going to quickly revert. And so what I'm doing is I'm just undoing the change that I've just made and what we will see in a couple of seconds here.

It's going to be that this is going to synchronize up and be clean again. So this is that the purpose of what ACM can do. But ultimately I think what we need to talk about is how we make that a little more rigorous for change management how it fits in.

So I'm going to do another thing here I'm going to show you what an ACM repository should look like and hear what I want you to focus on is quite a simple structure. You've got folders and then manifest inside of those folders that get applied and and what we can do here is we can look at any of our repositories and quite easily understand what's been applied to the system. So what I'm going to do now is go the proper approach of how I should apply a change in my system.

If I was an infrastructure software developer which means I'm going to open up a branch first thing and going to request my chain trade going to create a feature new namespace branch and I'm just going to go ahead and drag my new namespace out of here and add it there. I add another namespace manifest. And that's all.

Right now I'm going to do this now because the CI will take a little bit of time and then I'll show you the architecture diagram to make my commit I'm going to push that up. Now I haven't yet actually told any of my team that I want to commit this change. I could still be working on it right now.

How I do that is a trigger the merge request. Now this is the area where I can run some automated test form a code review. Now typically in a in a proper change management process it wouldn't just be me myself that has access to all of these repositories.

So what I'm going to do as the person who is the authority is I will tell it that when the pipeline succeeds if everything is good and we have a bunch of checkmarks I'm going to merge that in and I'll show you why. What I want to do is jump back to the architecture diagram here so we said we needed something a little bit more robust to manage change as it applies to gitops and so what we've built out here for some of our customers looks at how to manage the effect of maybe negative changes across an environment or multiple environments and how to do things like perform separation of duties. I.

tests that will that will take place. So we're going to combine all of our configurations for each cluster we're to validate the syntax with it with a Google Anthos tool called Nomost and then we're actually going to also validate that any of the policy that we've added to that system is going to go ahead and be successful as well that we haven't created objects that make the clustered uncomplaining ones. All of those actually have checkboxes beside them then we can actually approve the merge emerged and once we've merged that code then we're gonna do something different.

We're actually going to take all those changes and create new branches and merge requests automatically against our development and production environments and we've separated these out into their own repositories and projects within GitLab for a reason. The reason here is that each one of these environments has some CI testing as well that might need access to different systems in different keys and sort of secrets right. And at the end of this once everything's deployed, we're also going to go query the cluster for the status of the deployment.

And so we don't want to mix development keys and production keys in the same project. We also have a separation of duties between the operations teams who can actually an act change on each one of these environments. So here we've built really what fits in with the change management process the good separation of duties well we've automated most of the flow and again simplified how we can manage cluster configuration for many many many clusters.

We're gonna do is jump back into our GitLab tool and take a look at the status of our systems so far. So we can tell that we've actually got the pipeline the first pipeline the data validation was successful we rendered our can fix we actually formatted our content and we did some validation and if we take a look at validation we had our policy controller validation and our validation with no most of that. So we were really really happy then with just running the last stage here of our of our committee.

So we're actually going to go ahead and apply those changes into the appropriate repository and I think we take a look right now should almost be done so that's actually open up the merge request. If I take a look at the repository that's now owning the configuration. What I can do is actually take a look at the open merge request and here it is.

So I could be a separate team but now I know exactly what changes are being proposed which is an adding a new namespace. Any conversation about it and then I'm going to click merge as this happens it will be automatically applied to my cluster. And at the same time I'm going to kick off a bit of a pipeline jabbering and to validate that the pipeline was actually going to be successful.

So what this does in the Anthos world is with the nomos command and connecting to the cluster and checking for the state and making sure that the commit hash is actually what it should be. You can also see here that within this merge if I had made a mistake I could quickly quickly revert. And again I will be back into the previously known configuration so I can tell that I'm still sitting here which is great.

I'm going to go ahead and just double check that I had I new namespace created so I've just added a bunch of vigor around that application. But most of it was automated and I've safely applied this change to a lower environment, separating that change and the process around it from my higher environment . So as we jump into the last couple of slides as we talk about and start applying gitops in our organizations there's a few recommendations here.

Things like using protected branches which means making sure that only certain team members or certain people have the authority to push to an appropriate branch divine a defining good secret management strategy strategies is going to be important so that we don't cross pollinate and and have access to keys that people shouldn't have access to as well as leaving all of our keys out of our git repository system as far as code goes. Making sure that we're doing frequent code reviews and also frequent automated tests again making sure that we understand what's going to happen once we make the change. Removing the human from the production system as you saw there in that system.

I didn't actually have access to make the change. I only had access to make the change through the gitops process through GitLab CI. And then finally monitor the status of the automatic deployments.

Just because we made the change automatic and be tested isn't a guarantee that it's going to be healthy. So let's make sure we go ahead and monitor that status just again. A quick recap recap.

If you're going to take a look at gitops principles, right, we want to describe everything declaratively and we're going to do that in git and in GitLab. We're going to version every change never a full audit trail. We're going to automatically apply the changes to the desired state and we're going to alert or validate that everything is the way it should be.

With this fixed for us as it sped up time and increased stability of our system but it also allowed us to simplify our rollback processes. So if we put this all together we can say that Anthos and ACM allows you to start walking into gitops but you're going to require tool like GitLab and GitLab CI to help with the change management approach of how gitops fits in with your organization so that's all I had today. You can check out the contact information below if you have more questions.

Thank you very much. I am it with Arctiq. I appreciate your time.