How to Build a Better Pipeline: Using CI/CD For Better Infrastructure Management and Automation – Techstrong Con 2023

Hello everyone and welcome to Tech strong con 2023. My name is Adrian. Elisoo.

I'm a senior developer Advocate at Cisco and I'm very happy to be part of this conference this year the session that we're gonna go over today is called cicd pipelines for infrastructure automation. And let's Jump Right In it. The agenda for our session today is we're gonna go over and have an Overlook at cicd what it is how it came to be over the years after that.

We're gonna have a look at gitlab, which is the Version Control / pipeline component that we're going to use for our for our setup. Then we're going to go over a demo. I'll show you an actual pipeline that works based on gitlab and other products that they all come together to actually build a cicd pipeline for infrastructure automation for being able to configure in this demo will configure a couple of Cisco Nexus virtual switches, but it could be anything else.

Right? Any other type of infrastructure can be configure the same way and then at the end we'll wrap up with some resources. I have some links for you if you want to follow along and and get some extra information about this presentation.

All right, so ci/cd overview. That's the pipelines have been developed over the years mainly in a software development environments in order to be able to faster less error prone. and and just being able to bring software and software features faster to a production environment.

Right? So what it used to take developers and development teams, you know weeks if not months to develop a feature to go through a build process give it to the build teams and give you to the QA team. then, you know try to deploy it in a testing environment and a production environment.

So all these steps. have been automated throughout the years right so CSD pipelines came to be part of that automation process of developing and producing and putting into production softer quicker as their and with less errors and less bugs. So depending on what?

Where you are in the pipeline, you could have several different parts automated right? So the simplest pipeline would be the CI part in cicd and that stands for continuous integration. And this basically is the practice of integrating code that developers are producing.

So usually you have a team of developers, right? That develop a certain product a certain feature and usually use separate branches. And they at the end of the day or at, you know, at the end of the week they submit they basically do a get push.

They push the softer changes that have worked on. Right, they push it to a central location. So those branches get merged and pushed to a central location a central server somewhere and initial testing starts at that point right once we have all the software submitted in central location You can start doing some initial testing make sure that there's no bugs or catch them as soon as possible.

And then there's sure that you know, the quality of the code is good and then there are no integration issues. Right? So continuous integration this stage just consists of gathering the software and the code that the developers are building and all having it in the central location and Performing initial tests making sure you catch bugs as soon as possible so that you know continuous integration as part of this pipeline.

Now if you take your pipeline to the next stage, which would be you know, the CV in cicd which stands for continuous delivery. This takes the software development process a step further And it creates a release what we call right? So we have the code in the central location.

You've done initial testing. You've caught bugs you make sure that there's no integration issues and then you actually build and create a release out of your software. So this could be you know, creating a binary or creating a jar file.

If it's Java code at your writing your application in or it could get darker container that we Python and making sure that pythony's running in there and exposing and running your application correctly. So whatever the release is in your specific environment it that gets created at this stage. You run additional tests and checks and of course all tasks are automated.

Right? So everything so far part of the CI and also the cd part of this is all automated. So releases are created you do so additional testing and you can also push it to a staging environment, right and one additional tests at that point making sure that the application performs and the features that you've developed perform as you are expecting them to Now if we take the pipeline a step even further to another CD part, which is continuous deployment.

This takes the delivery step further, right? So you've build the software that you've that you've developed you've tested it. Now you want you've deployed to a staging environment.

Now you actually want as a continuous deployment stage to actually push that to a production environment put that software that feature that application whatever it is, you're gonna push it to a production environment. So that happens at this stage again, everything automatic. There's many different deployment strategies available out there.

You do blue green Canary rolling and this is basically just part of the infrastructure that's running your production. Is it, you know half of the servers that are running get the new version the the other half are still on all version. So there's you know, AB blue-green all these types of different strategies to deploy to production environment.

So this is basically CSD pipelines and how they are being used. By software developers to build software to improve their practices and to make sure that everything gets delivered as fast as possible as quick as possible and with less bugs and issues as possible. So how can we take all these lessons that we've learned over, you know, 10 15 years and in the soft development business and applied them in the infrastructure automation space.

So, I believe we can and I'll show you in the demo what we mean by this. Association automation means that our configuration changes are done through automation. So automation here means, you know as a tool for configuration management such as sensible or terraform.

Could be used to automate the infrastructure the configuration changes. You get a historical insight into the changes made to the network because you keep a version control of all your changes. Similar to how softly developers build branches and emerge them and they have a clear history of all the changes.

They've been performing over, you know, what days and and weeks and months same thing with infrastructure. Now, you have the option of keeping track of all the changes that have happened over the network based on the CST pipeline that you've built. You can help goers also validate changes prior to pushing the configuration into production Network.

You could have a test Network and that's what we're going to see in the demo. I've built the test network based on CML Cisco modeling labs. So you could have you know, your changes portfolio test environment make sure that there's no breaking changes that there's no unforeseen issues that are being introduced by the change in the network.

And once you have that you can safely push it to your production Network very similar to what we've seen we in software. And then you have the optional so if an issues introduced to the network to roll back those changes to a previously known good state. So if there is you know a change as you perform is creating issues, you can quickly very back to the previous version very fast and easy and minimize that downtime or that problem within your infrastructure.

And of course you can take integration. To the next level by integrating your pipeline with change and ticketing systems such as service now or an iPad solution or you know such as netbox. and you can truly have a continuous deployment type of Pipeline and set up for your infrastructure at the same time.

All right. Next let's go and talk about gitlab, which is a complete that was platform. If you don't know by now, they also have a Community Edition that you can install and run your own Enterprise GitHub environments on Prem.

com SAS offering that's actually the one I'm using my demo. I've used the CE version of gitlab. So it is a complete DeVos platform, right?

It can be used for project management. Of course, you have host of give repositories. You have Version Control that's their bread and butter.

That's how they started but then they added extra components on top of those given posteries and Version Control and they have a built-in cicd component that you can actually build pipelines using gitlab. And pipelines within gitlab are fundamental building block for cicd. Right?

So we see here on the right hand side. We have a several pipelines. We have four that are running at the same time right here, right?

There are different stages. You have a couple of them that have been canceled you have two of them have passed at the bottom here. so Pipelines within gaze lab Are composed of jobs which is you know, what should be done as part of the pipeline and stages when you should run the jobs.

You know what the pathline should do. A typical pipeline consists of four stages so we would have the build stage which is like your compiling something development world. We have your tests stage, which is you do your unit testing at that in that part.

You have the staging the point of staging environment you're doing integration tests extra tests in there and then last a lot but not least you have your production stage where you deploy to a production environment? All right, so this is kind of like a typical pipeline in a software development world GitHub has the concept over render, which is just a software process that is used to run the jobs and send the results back to gitlabs. tml file, that's where you define your pipeline your stages.

Your jobs enough everything comes together in the yellow file. So when the pipeline is trigger, usually by You know a good push to a central location that triggers the pipeline you type sees that there is being changes that have been performed this specific file that's being monitored. gml file which contains the definition of your pipeline.

The runner can implement the variety of executors. So this is just how do you want to run the jobs right in which type of scenarios you could have the runner spin-up virtual machine as part of using parallels or virtualbox? You can have it spin up a Docker container for example, right and run.

The test and the pipeline the docker container and that's what we're gonna see in our demo. That's what I've done or you could have Iran shell commands or you know, you could have it Associated to a device and once a certain set of commands you can have it integrated with kubernetes or you know, it could have it custom I could have these Runners customize as much as you want and do exactly what you want as part of your pipeline. So pipeline.

gml file. So you have your example of a pipeline with three stages right a build test and deploy. It uses a Docker the Alpine Docker image and then you have here several different jobs of what you should do at each stage the build stage.

There's a couple of scripts that are being run the testing stage. Just echoing some text the console same thing with the deploy. We'll see in our demo in a bit how exactly this gets configured for an infrastructure automation type of setup.

All right. So now let's go on and jump on the demo part of our presentation. So our test environment today.

I have a Centos hosting server that has gitlab install on it. So that gitlab Community Edition is installed on my Centos server. The dev workstation is just a Mac computer.

My network simulation. So like I was mentioning is running on CML system modeling labs. This is a test environment.

I have several different devices in there. I have a couple of switches a couple of routers. They're interconnected.

They have ospf running between them and we're gonna actually update that ospf configuration as part of our demo. The changes are being performed by sible in the background price on the pipeline. Actually runs a niceable Playbook as part of the configuration stage of work for SPF.

Gitlab is our source control and our pipeline. Main postery right and then we're also using Cisco pyths which is our testing framework from Cisco and we're using pyts basically to take snapshots for infrastructure before and after the configuration change is performed. All right, let's jump in and go over our demo.

So I have here by CML environment. So you see here Sysco modeling Labs. I was saying I have these on the bottom here are two.

Virtual switches. Next is 9000 d I have a couple of CSR 1000 views which are Cisco routers virtual routers here at the top. a couple of hosts such as Linux machines So I want to make sure that you know, the configurations are already applied.

Ospf is running is an ospf is just you know a routing protocol. Is running between all these four devices. And as part of our pipeline, we're just gonna change that configuration and add one more or SPF interface and and one more route on each of these two.

Next feature switches here at the bottom. So I start the demo, this is the test environment you're seeing running on CML. Or just have a quick look at our switches in a bit just to see the configuration before.

We apply any changes. This is our local installation of gitlab C right running on that Centos. gmo.

So this is that yellow file that contains the definition of my CSD pipeline. I have my pipe. Yes folder which contains my pipe yes tests.

And test files I have my post bars and who bars as part of my ansible configuration, right? I have my Docker file I might as well that's crg. And then I have my hosts here also sparked in my answer configuration.

Am I config principal and then I have actions here is basically just what actions will be performed contains files that are being used as part of my pipeline. So this is the cicd file right folder. Sorry that has all these folders and files.

and this contains this definition of my pipeline which Right here. As you have three stages as part of the web pipeline, right the pre snapshot with paid. Yes.

I'm taking a snapshot on my infrastructure all the SPF routes. I'm making sure that they're all there prior to performing the change then I have the deployed ospf. Right stage which is configuring you can see right here.

You define the script as part of my new floor SPF stage and my ansible playbook that's running from the actions folder. ml file. Which basically it is ansible playbook for configuring ospf?

in the free snapshot stage Of my pipeline. iemo. So that's exactly the CML test, but that you've seen previously.

triggered that I file that's basically what ideas will be looking for. But those ospf labor shapes the browse to be over there. That's where everything is defined in tml file.

What piety has will actually take a snapshot of and what exactly is containing the snapshot. thing the snapshot into this to files right here I'm running like I was saying a Docker image. So all of these pipelines everything is being performed in this Docker image that's coming from Dr.

Hub. com you'll be able to see I have a Docker image over there and it's getting pulled down as part of the pipeline. It's pulling down this docker.

Image that has pre-installed all the pre-required components. So I had by yes. We installed in there.

I have Docker I have also Ansible pre-installing there. So all the prerequisites are already installed as part of that Docker image. So that's where my pipeline will run although it clone into that Docker environment and then trigger all these stages one at a time.

So we have the presnapshot. Right, we have after that the deploy SPF stage where we're running the ansible Playbook. And that's exactly the command that's being run as part of the script and the stage at that part and then we have the post snapshot with piets which same thing right changes directly by TS one of the pythia pts jobs from there specifying test file and also the trigger that I follow for pulse trigger and that file is important because we want to make sure we edit that to capture the new changes that we're pushing with with ansible.

All right, so here I'm connected to My remote repo I have the whole Bars by two switches 177-178. This is the SPF configuration that's currently existing on on the switches. And let's make sure.

We have it configured we have already a loopback 100. Right, and we're just going to configure a little back 101. So that's just one more look back interface.

Just gonna add it on both of these switches and this is just adding one extra routing interface, basically. On both of the switches and it's getting propagated. Throughout ospf throughout to all the other neighbors and all the network.

So we see here before the change. We have that little bit 100. There's no little bit 101 and that switch to and I'll switch one.

Same thing. We have just pull back 100 and let's say no loopback 101 that will be part of our pipeline as we pushing. Or SPF memberships is established.

All the devices are pre-configured ospf memberships are all up and running. They're exchanging routes. So now if you go back to our configuration for our switches and like I was saying important parts here.

It's to save the files and also edit our post trigger data file. So this is the pipe. Yes file.

Right that will check. To make sure that the new routes are being learned by both devices. So switch one basically learns the route from coming in from switch to and switch to learn the route coming in from switch one that dot one and Dot two so basically both switches.

We want to make sure that the show IP route we have these new route as part of our host checks. So that's our trigger data file for piot. Yes.

We want to make sure that we save that so now we're happy with our configuration. We do a git status. We see that have been changes performed on our both of our switches.

We have those new low back interfaces configured also part of the trigger data file. We have changes in there. We don't get commit.

Added loopback 101 and then of course, we don't get push. And this will basically push the changes to our central location, right? Login as part of our first the remote gitlab instance.

There we go. The branch has been pushed. And now if you go on the pipeline automatically we see that the pipeline has been triggered right so we have the three stages.

We see here at a Loop 101 as part of that pipeline, right? Yes. We snapshot now is running that stage of the pipeline is taking a snapshot with piets.

It's pulling down the image from Dr. Hub. Right?

It's good cloning the repo into that Docker image. And it's basically after that running the three stages configuring. We go.

So we see that rspi is configured. All the routes jump has successfully completed and this stage so stage one. You see here with the Grinch with a green check mark right is successfully completed and we're moving on to stage two, which is the plowing ospf.

configuring those loopback 101 on both switches President switch one switch two all by 101 being configured by ansible on both of them. and this is using our CSV pipeline so there's gonna take a 15 10 15 seconds for ansible to actually get that as simple Playbook go connect to the devices configure the low back interfaces. And make sure that those changes get applied.

There we go. So job succeeded on both switches. There's new interfaces one-on-one that one and one one that two right.

There will be 101 configure. So these are part of our ansible. Stage we see here stage two deploying rescaf has been successfully completed and now we're moving on to stage 3.

And we see at this stage that now we have low back 101. Is where you can figure over there, right so it wasn't there before. We check that is in there and in the background also, you see that post snapshot High successfully being completed.

We know now that the routes have been propagated and learned between the two switches. It's a little bit 101 on switch one has been learned on switch 2 and vice versa same thing. Go back 101 on switch two.

Has been learned on switch one and switch. Want to go back on 16. There we go.

Try Prado SPF. We see right here or switch one. 2 right there learned so that make sure that that Post configuration pipe.

Yes snapshot we see it's come completely successfully. So the pipeline ran successfully right? It's fairly simple always get configuration pipeline.

You could adapt this. You see it passed it took. About two minutes and 13 seconds to complete, right?

so this you can definitely see here how easy is to configure infrastructure by using this pipelines, right? So you have a test environment successfully configured. and then you can go ahead and Post this production environment, right?

So you're happy with the change that have been in the CML environment. You can go ahead and make sure. You you have it also performed in a those changes in a production environment.

so resources I have here a couple of links. So be on the lookout. I'm writing I have four.

Blog posts around this pipeline. So we're starting to build this pipeline from zero. I walk you into a series of blog posts through building this exact pipeline from zero from Starting Gate lab on Central as customize your Docker image from installing pts from building the pi ideas test bed and test files to building the ansible plan board everything.

com To look for my name. Be able to see all the blog series. net.

com be able to to clone its adapted for your own environment. And that was all I had thank you so much. I hope you found this presentation useful.

I'm looking forward to hearing questions. I'll be part of the virtual events and listening to any questions that come in looking forward to your feedback. Thank you so much super happy to be part of this conference and looking forward to the next one.

Take care of everyone.