Christian Hernandez – Pipelines and GitOps: What’s the Difference?

Hi, my name is Christian Hernandez. I am the developer experience lead over a codefresh. And today I'm going to be talking about pipelines and get UPS.

What's the difference there? Um Real quick. I'm just gonna go over a brief agenda what we're gonna talk about today.

I'm gonna be talking about cicd get Ops and how they both fit together in this new Cloud native world. So so first let's talk a little bit about cicd, right? The crown jewel of devops is I like to call it.

So I always when I talk about CA City, I always think back to the agile Manifesto. I think really that's where it all started and devops really appeared right after the agile Manifesto and it it came up as a sort of need to to the fast moving nature of agile and scrum right? There was no really definitive date of when devops kind of appeared but it was a you know, it appeared right around 2007 and it was really um A thought process from both software developments and it professionals right from both sides of the aisle talking about how really they've had it really had enough the traditional software delivery mechanisms and kind of how it it was, you know, it was a dysfunction inside the industry and something really needed to be done so developers and it professionals.

You know often have competing objectives. And you know, they have separate kpis. They have, you know different the metrics to which with the judge success and really the the operated really in silos and devops really is about bringing those teams together and you know breaking down those barriers because a lot of teams adopted agile, but we're still having trouble with like releasing software because they had this new methodology that's out agile and scrum but they still had kind of like old ways of developing and deploying applications.

And so the idea behind devops was to kind of like really let's let's break those barriers down. Let's make sure that we have a process that's complementary to Agile. So um, and really this is kind of like the It gave birth to like the cicd right?

See I see is really the devops tool belt is the crown jewel of devops as a as I mentioned earlier and and see I see came about to really Meant agile and scrum to continuously release and and develop applications. So there's really three parts to it, even though it's ci/cd. There's really three parts to it.

And then they all build on each other right where the first part is continuous integration. Whereas the idea of you continuously code and continuously commit very often and you test against those commit developers that are practicing continuous integration are really committing and merging in back to the main branch very, you know, as often as possible and and those changes are validated right those test Suites are always continuously happening, right? So it's the continuously integration you integrate to your code continuously and so continuous delivery kind of builds off of that aspect was like well if if we're continuously coding and we're continuously integrating with Um with with our code base, let's just also this deploy the Devon test right?

Let's not just commit code and let's not just make sure code is always up to date. Let's deploy that so we can see those changes in an environment. And so that's the idea of continuous delivery.

It's like let's deliver that code that we built into into these Dev in test environments automatically so that way you know it we can see those changes so and then continuous deployment builds off of that really and this is really the scary one as I like to call. It is the continuous deployment. It's like well if we're automatically deploying to Dev and test if all tests are passing and all, you know regression tests and all all these testings and and and user user tests are passing.

It's just deploy to production right and that's kind of really like the Crown Jewel the the Nirvana right of cicd and devops where it's like, let's like Auto commit, you know an emerge into a branch could automatically end up in production and really Cicd, is that that? Aspect where you you have continuously committing continuously merging these changes into the environment which end up in production. So so that's kind of like the ci/cd aspect of it and with ci/cd a lot of tools emerged in this aspect in this world.

You have cscd tools like like Jenkins, right? So Jenkins being the most popular ones that are specific to cicd right Jenkins being like that platform the most popular ones but also other tools that came up during this the building out of ci/cd and devops and agile, and these are infrastructure as code tools, you know, just like development had to adjust to the devops methodology operations had to as well and so they had things like puppet chef and simple those are kind of the tool sets that emerged to help the operations meet that that that demand for acceleration that demand for continuous integration continuous delivery and deployment and so most of the time the the cicd platforms like Jenkins were using conjunction with infrastructure as code tools. In you know in their respective environments, right?

So they've been using that in tandem together. So so you have the the aspect of okay I have You know code. that I commit continuously and that gets deployed automatically, you know using tools like Jenkins in conjunction with infrastructure as code tools, so You know, the the automation is not on our own place, right?

So, you know what what else is needed after that, right? And so and so many devops practitioners have been hearing about getups. Right?

And so what is get UPS, right? And how's that differ? Right?

And so, um, it's kind of take a look at get ups and what is getops exactly so Um get UPS has been described as a developer-centric approach to continuous delivery and infrastructure operation, right? So the idea is that you use git as a source of Truth you do everything as code. And you manage the system via pull requests, right?

So they call it the operations via PR or um management through through get that sort of operation management get so this kind of idea and notion was birth in the during the whole container kubernetes Boom the as the industry continue to evolve so they practices and you know, kubernetes became more and more ingrained and became more and more mature many workflows adapted to the declarative nature of kubernetes. So people were starting to Storing and versioning kubernetes configs in git and ci/cd deployments turn from Custom scripts to kubernetes deployments, right? So no longer.

Whether custom scripts or you know custom infrastructure as code manifests at the end. It was basically kubernetes manifest and the kind of migrated over to that aspect row, so So this kind of this kind of sounds like things you're probably already doing right? So it's you're you're thinking okay, you know, I'm doing devops, you know, we're storing everything and get we're averaging everything and get you know, we're deploying to kubernetes like, you know, why is is so let's get ups and actual thing or is it just the buzzword?

Right? And so the answer to those both those question is yes, you know when something is a buzzword, there's actually truth behind it. And you know, it's also an actual thing and in standardization is always a good thing so around November 2020 Amazon codefresh GitHub Microsoft red hat and we've works, you know, all the big players in the cloud native space announced.

The getups working group, right? So what is what is the great Ops working group? It was basically a working group for interested parties to get together to Define.

What getops actually is right. So there was a lot of you know, it started as a buzzword. There was a lot of interesting players in the cloud native ecosystems saying, hey, let's actually get together and Define what get UPS is and is meant to be kind of a vendor-neutral standard to Define get-offs.

Right? So kind of like a vendor neutral like this Define what get UPS is and create that team arcation of What It Is What It Isn't So and this kind of bootstrap itself this this gaps working group would shop itself in the application Sig the application delivery Sig inside the CIA cncf and you know in October 2021, you know after you know a year working together the gaps working group released the open get up principles, right? And so the open get off principles.

set a printables principles that Define not only what gitops is but how What A system that is implementing gitops looks like right and so these principles are there's four of them. Right and they're meant to be generic but But also descriptive at the same time so so A system that is operating. So that is operating on getups and Whether or not you know, you're doing gitops is um, you know, first the system has to be declarative.

Right? So system managed might get up must have its desired State expressed declaratively, right? So that kind of Goes to what is happening with kubernetes nowadays, right?

So that shouldn't be a surprise, right? Okay. I have yaml I'm storing it declaratively, you know that the statistic is declarative.

Right? So we're leveraging kubernetes for that declarative aspect of it. But then we go to the second aspect of it.

Whereas is Virgin version and immutable meaning that's a desired state is stored in a way that if forces immutability And versioning retains its complete version history, right? So that this is the get part of get Ops notice that it doesn't doesn't say get explicitly. It just says that it needs to be immutable and it needs to be versioned.

Right? So for example, you can use the S3 bucket. to store those you can use any other source control management system and obviously get Is a fits within the within that Paradigm and so the the third one is that it's pulled automatically meaning that the software agent automatically pulls the disk the desired state.

So this differentiates it from ci/cd and this is where it starts to to pull away from traditional ci/cd to more of a cloud native ci/cd would get Ops is that it's pulled automatically meaning that there's no there's no like web hook that triggers something to apply the change. There's a there's a software agent sitting on the system automatically pulling those changes and then and then Lastly it's continuously reconciled. Meaning that software agents continuously observe the actual System state and the declared State and will and it'll attempt to apply that the the desired state right?

dev. diff to learn more. So, how does How does gitops actually work right?

What does that workflow look like right and so I'll get Ops is rooted in kubernetes. So it's it's always you know. People ask me like am I doing get apps am I not doing gitups is like, well, it's rooted in kubernetes because githubs is it mirrors the closed loop control theory that kubernetes was built on and so meaning that there's a desired state.

And then there's a running State and then the controller just attempts to apply the the desired state so user commits a change to attract repo right? So I commit a change I have a git repo. That is that's storing some yaml right some kubernetes yaml and I make a change right whether it be the image tag that I'm using or the I'm using the the replica count.

I changed something like that. I make a commit to a specific Branch. So here the GitHub school is continuously.

Looking at the get repo and once there's a change. It applies those changes to the to the trip to the kubernetes cluster, right? And so and not only that it's always continuously monitoring.

The kubernetes cluster it's looking at the currently running State and if there's any sort of drift, right you can configuration drift it'll attempt to correct those those drifts as well. Right? So here the only thing that happened is I made a commit and in the github's tool does things asynchronously for me.

It'll it'll say oh hey there was a change in the get repo. Let me pull those down and apply those directly to the cluster and so So get Ops versus what I like to call traditional ci/cd, right traditional ci/cd is event based. So there there's an event that happens right?

I make a a commit to a repo which then That will then has a script that triggers a build which then at the end of that it triggers something else, right? There's an event that happens that kind of like a domino effect, right? Then that goes down and get Ops is based on a reconcile Loop.

Meaning that there's there's nothing triggering that get UPS tool to do its thing. It's just continuously monitoring, right and I always I always like the analogy of a thermostat, right and this goes back to the control theory. This is something that is used to describe the closed loop control theory.

So think of a thermostat versus having like a wall mounted air conditioner so you get warm right and you have a wall mounted air conditioner that's an event I get warm. So I'm gonna stand up and turn on the air conditioner. And then another event happens like well now I'm too cold.

So I'm gonna go stand up and turn off that wall mounted air conditioner. There's an event that happens. I have something happens.

I actually have to get up and do something in order for something to happen, right? Whereas a thermostat is more closely related to to get Ops and kubernetes right with that closed loop control theory is that I just set my desired State. I just set the temperature to 72 and the thermostat continuously monitors the temperature and it'll turn on and turn off by itself.

So that's kind of what where you have to think about the Paradigm between traditional cicd where an event happens. I get warm. I stand up I turn on the air versus get Ops where I set my desired State and the thermostat takes care of turning on and off the air conditioner.

So another thing to note is that get Ops is CD only right continuous delivery continuous deployment only has no No, no concept of CI right building something or continuously integrating something. That's not its job gadopsis CD only so so get off the CD only. We're just get Ops fit in so, you know when looking how get up approaches ci/cd you first have to understand that.

CIA and CD aren't aren't aren't necessarily as tied together as closely as you think it is, right or is closely as you as traditionally that it is, right. So people usually glomcey together because popular devops tools like Jenkins aim to do both. um, and you know Jenkins successfully created a platform to manage both CI and CD so, you know people think of it as a one thing as one management system, but what you know with kubernetes microservices Cloud native architecture, you know, you know those type of paradigms A lot of a lot of a lot of folks are starting to decouple those two ideas, right?

And so so there's some challenges with that right? And so there's CI if you think about it is primarily a synchronous process something kicks off and all those all those steps that happen in your CI system our synchronous, right? It has a start it has an end and it's usually triggered by some event whereas get Ops conversely is asynchronous, right?

And so You're trying to integrate a synchronous task with an asynchronous tool and it could be challenging right especially if you're trying to integrate system that haven't been in place for a while. Right? We all have Tech debt, you know you we all have things that have been in place and are working for a while and trying to adopt something new that is different could be challenging right?

And so there's a few ways you can get started. I would say in your gitops journey and fitting in with your ci/cd system. Right.

So the first part is what I like to call CI managed, right? And so this is this really is. Like what you have now, if you're not if you haven't adopted githubs already is this essentially what you have now, and it's essentially the CI system owns, you know, or the pipeline is I'm gonna use them synonymously in this case.

Owns the complete deployment right? I make a code change application gets built a container gets built. It gets pushed to registry as some sort of deployment rollout happens.

And so, you know, the getups process is kind of like an afterthought. It's usually done with the floating tag, right? So like, you know, they get up system is looking at your deployment configuration and you know part of the push to the image registry.

Maybe you have a task that tags it for production, right? It does a floating image tag, right? So it does kind of like a Docker tag, you know, whatever your images colon production.

and then, you know the get UPS system kind of looks for that and reconciles that right and so the the benefit of this is that it kind of follows traditional ci/cd and it reduces your need to architect but The biggest drawback and especially if you want to adopt get offices. It's not really get Ops, right? You know the idea of using a floating tag is you know, you're using to get as a source of Truth and someone can force push attend.

An image tag, right I could force push something to be production right in all of a sudden, you know with that tag production all of a sudden your production changes, right? And so usually floating tags. I would say is bad practice or something you shouldn't be doing in production, but This is step one in a journey, right?

This is usually people start here low barrier entry, right? I just have a get-offs controller that is looking for specific tag to deploy to production. So um the next aspect of where you can integrate it is you have a CI owned and this and you do CD video get ups and this is where the the CI owns the process but The get-ups controller owns that actual deployment of it, right?

So it's essentially it's a handoff as as I would like to say, right you've kind of fire and forget where you You make a commit. It does the bill that does everything it needs to and then this instead of the CI. Um, you traditional ci/cd instead of actually doing a deployment the CI pipeline system kind of hands it off to get off.

And so great thing about this is that it's the get UPS principles. It's compliance to get Ops principles and it also keeps the linear aspect of your pipeline, right you think of like pipelines being kind of like linear it keeps that aspect of it. And so it's like easy.

It's easy to adopt, right? Kind of the drawbacks to these is that it adds a layer of complexity. Right?

Like you're now you're mixing specifically a synchronous process with an asynchronous process. And you know, now you have like two systems to kind of manage one that builds and one that deploys. I know this it's a paradigm shift.

Now where everyone is used to having ci/cd be one thing and one platform now you have to right and so it kind of adds layer of complexity by mixing synchronous process with an asynchronous process. And another thing is requires refactoring, right you just have now another since you have another system you need to account your pipelines need to be account for that other system. Hopefully doing a thing right?

It's kind of the the ideas like I hand it off but since it's an asynchronous process, I don't really know what happened after I hand it off so You know, that's kind of the CI owned and CD. They'll get ups and then also this is kind of the The Nirvana of it all is the CI triggered but get UPS own, right? And so this is where the CI builds the application and then get off kind of handles the rest the deployment, right?

So the CI just all it does is just builds the application and then get UPS handles deploying that massively into your into your environment and the CI interfaces with getops via automated P automated PRS, right? So now you're you're Now your Source control management system or get right? I'm gonna use an interchangeably here.

Becomes that interface between the two systems so you can do this either gated or not gated right? You can either do automatic commits to whatever branches you're tracking or you can have a gated by a PR right? Someone can manually look at the pr and you know, click merge and then you know, that's where gating happens in the pr aspect of it.

Um, you know, the the benefits of this is now cicd are completely decoupled, right? So they're they're independent of each other and have independent jobs. It keeps CI and CI and CD and CD.

They don't know and you know, they don't cross pollinate, right? And so a big drawback is I just obviously no longer a simple linear process, you know you can have you know tests trigger other pipelines was just then triggers other gitups processes and in your different environments, you know, it may be a little bit difficult to troubleshoot because you know, you don't know which thing triggered which which reconcile and also there may be Um pipelines being fired for other reasons outside of getups, right? So they're you know, people running pipelines manually that's totally valid and you know, totally something that will happen and your processes needs to account for that needs to account for the automated and non-automated kickoff of CI and so there's a paradigm shift right?

So really when you're thinking about pipelines and get Ops is that you need to keep in mind that you're no longer promoting code right you there? There's a paradigm shift. By you know traditionally you're thinking I'm promoting my code from one environment to another.

But you're not doing that anymore not with gitops not with kubernetes. You're promoting manifests right A lot of the times what you're promoting is a simple yaml configuration that points the new version of the image and you need to change your change your way of thinking about like hey, um, I am promoting a deployment from one environment to another right deployment yaml meaning when I say deployment the actual yaml manifest. So you need to kind of like make that paradigm shift and so um, you know, that's one of the the many things that you need to keep in mind is that you know when you have the traditional Pipelines With you know, you're moving to a getups.

Way of thinking you're adopting kubernetes is that you're okay, you know, I want to top get-ups, you know, my code is being built. What's getting promoted into the next environment is a yaml file? Right?

And so, you know, what are the many things here at code fresh that we're building is a host to get UPS platform. So, you know, I invite you to try it out right host to get Alps platform is built on our go CD. And it's essentially hands-free get UPS management.

So it integrates with existing pipelines, right? We have support for GitHub actions. You'll support for Jenkins, you know, you plug in basically your your pipelines and Not only do you no longer you don't need to refactor those pipelines.

You also have a hosted gitops platform where you don't have to manage that right? So you can stop scripting you start deploying onto the platform. So I invite you to try that out.

It's free for small teams and open source software to use host to get Ops platform on code fresh built on Argos CD. So with that I want to thank you for listening to this presentation, and I hope you enjoy your day. Thank you.