DevOps Building Blocks, Part One: How CI/CD Drives DevOps Automation – DevOps Unbound EP 38

Hey everyone. I'm Alan Humel, c e o founder, editor-in-Chief for Techstrong Group, and you are watching DevOps Unbound. DevOps Unbound is a twice a month or biweekly, uh, video series that we do here on Tech Drunk tv, and it delves into various aspects of DevOps.

We've been doing DevOps Unbound now for more than two years, and we have quite a library of of episodes. You can catch those if you're interested on Techstrong tv. We have, I think, every single one, including the live round table versions of DevOps Unbound that we do about every four to six weeks.

And in that we have live audience participation as well. Today's episode, however, is not a live round table. It's with an outstanding panel of experts, and we're gonna introduce you to them here in a second and jump into our topic.

But before we do, I wanted to just quickly mention that DevOps Unbound is a joint production of Techron Group and our good friends at t Tricentis, the worldwide leader in continuous testing. Tricentis has been with us on this journey since day one, and we couldn't ask for a better partner and, uh, sponsor to work with. They, they've really helped us a a ton.

So many, many thanks to them. Um, but with that said, let's move on to today's show. Before we jump into topic, let me, let me introduce you to our folks in, in the panel.

I wanna start off with a good friend of mine, but as he mentioned, we were talking off camera. I don't think I've spoken to him in about a year. He's Ravi Lockman.

Ravi is Project Manager at Harness. Ravi, welcome. Introduce yourself to the audience.

Yeah, Thanks for having me. Excited to be here with everybody. A great panel.

Uh, I'm Robbie Lockman. I'm a product manager at Harness. Uh, a little bit about myself.

I started out as a software engineer and kind of made my rounds into platform engineering and distributed systems. Uh, and now I'm into a product management, really like helping people get their ideas out of their laptop and into the hands of others, which is production. And so, yeah, really stoked to be in the panel and learn and chat with everybody.

Excellent. Um, next up I want to introduce you to Brian Cole. Brian is Director of Customer Engineering at Tricentis.

Brian, why don't you introduce yourself? Alright. Good morning everybody.

So my name is Brian Cole, uh, director of Customer Engineering is a very fancy title. That means that I'm responsible for evangelism and advocacy about the importance of, in particular, my specialty performance engineering. I've been in this space for decades at this point, uh, working with different vendors, and it was a fascinating journey, being in the quality space and seeing the rise of DevOps and how that impacted QA organizations and what the quality effort needed to do in order to become relevant again in that, in this new reality that we're all in.

Excellent. Next up, I'd like to introduce you to a first time guest here on our DevOps Unbound Techstrong tv. Uh, Jackie met Michelle, and if I mispronounce, I apologize.

Jackie, senior product manager at GitLab. Jackie, introduce you, tell us a little about yourself. Well, thanks Alan.

A couple of things have happened since that bio. My name is Jackie Porter. I got married and I also have been promoted to product director at GitLab.

So we've got Excellent, a couple big things happening there. Well, Congratulations and congratulations. So, um, a couple of things about my background.

I am currently a product director at GitLab over all things continuous integration. Uh, we, we support many pipeline workflows, but also packaging and, and everything pre pre-deployment. Um, but my background is actually in release management and engineering operations.

Uh, I spent years doing that and helping, uh, teams kind of deploy with, with Kubernetes. So really excited, uh, to be here and learn from, from the best and brightest on this panel. Fantastic.

And welcome and thank you. And again, congratulations. Good to make sure we update that information.

Yeah, we definitely will. On, on that. Alrighty.

And then last but not least, he's a, he's a frequent guest here on DevOps. I'm bound with us. It's our friend Adam Orlean.

Adam is with Dell. Adam, welcome. Uh, hey Alan, how are you?

Uh, yeah, I'm, I'm, uh, happy to be here again. Great to see everybody. Um, yeah, I, I work at, uh, Dell.

I'm in their, uh, infrastructure solutions group and, uh, in their DevOps in their DevOps group. I'm a senior director there. And, uh, we're really, uh, at this point we're building IDPs, which is, uh, internal development platforms that, uh, are running to scale and curate workflows for all of our 15,000 plus engineers.

Uh, and on, uh, on the side, I, uh, I'm a part-time professor at Boston University. Great to be here. Very cool.

Thank you. And thank you, Adam, for being here. Okay.

So that's our panel. Let me introduce you to my co-host, though, and he's, he's, uh, my partner here at Techstrong. He's C T O and Principal Analyst of Tech, strong Research, c t o of Tech, strong Group, Mitchell, Ashley, Mitch, if you wouldn't mind introducing yourself, and why don't you just jump right into today's topic as well.

Okay. Thank you very much. And again, congratulations to Jackie.

That's fantastic. All great news. Uh, yeah, as Alan mentioned, we, we work together at Techstrong and I cover kinda three hats, uh, between analyst and C T O and also doing media kind of things.

But my background is running development teams, software developer cloud, uh, moving to the cloud, transforming applications, uh, kinda living that part of the journey, both in product companies and IT organizations. So that's enough about me to that point. We're, we're, we're starting a new series here on DevOps Unbound, um, called DevOps Building Blocks.

And we wanted to kind of break down some different parts of DevOps. 'cause it's a pretty broad term, right? A lot of things get piled into to DevOps.

And of course the more popular it gets, the more things get piled into it. But at the heart of DevOps, most people usually start with C I C D. And there's probably some pretty good reasons for that.

The continuous continu, continuous integration, continuous delivery, um, deployment. Uh, so we wanted to kind of jump into that understanding why that's a good place to start. Uh, you know, no matter where you are in your development team, you may be very sophisticated about how you do continuous integration already, or, you know, that's something to kinda shore up.

And then we wanna talk about automation. 'cause that's the thing that lets you work at higher speeds, higher velocities and things like that. So let me throw, let me throw that out, um, in your way.

Free, free to disagree if you don't think that's the best place to start anymore. But, you know, why is CI part of CI I C d, uh, a a you a typical place, a good place for development teams and organizations to start? Anybody?

Yeah, I can, I can take a swing at that one. So when we talk about continuous integration, it's useful to think about what manual integration was before this. And it was really, I'm a developer.

Mitch is a developer. Jackie's a developer. We're all working on the same application code when we write different code and different modules and check it into our source code management repository, unless we're integrating it right then and there.

It just kind of sits there, uh, side by side. All of our code changes. And then several weeks later, we try and merge all those changes together and get what are called integration defects, uh, the, these crop up all over the place.

And it's difficult because weeks have passed and the mental state that I was in when I wrote that code has long since elapsed and been replaced by other activities. Continuous integration just means when I check my code in, it triggers a CI system, a continuous integration platform. Jenkins is a great example.

Azure DevOps, A W Ss code pipelines. There's dozens of them out there. And it sees my code check-in and immediately triggers a build and builds and assembles all of the source code and make sure that it all integrates properly together.

And if it doesn't, it kicks my code out and gives me an error message saying, your code is garbage, Brian, here's what you did wrong and why it's not integrating, fix it. And it can do all sorts of very sophisticated things like rejecting the code check-ins, and there's this whole pipeline of actions that could be predicated based on this one action of the developers checking in some code. So this ties into a lot of things like properly using your CI system, getting more value out of it.

Uh, everybody's pretty much got one at this point. Uh, tying it into complex subjects like your deployment engines and what the downstream ramifications of that are. There's a lot of moving pieces to this, but fundamentally, that's the first place to start because it addresses a whole swath of integration challenges in one step, um, all in the developer space.

And developers love this. Uh, there's nothing better than being told, okay, so I have a problem, I can fix it immediately, as opposed to getting the exact same message four weeks later. Absolutely.

I I I'm sorry. Go ahead, Adam. I was just gonna say, and, and, and the cognitive load of development is, is very, uh, critical there.

Uh, you know, fixing something that you have full context in versus, versus trying to fix something, uh, six weeks later, right? Is, you know, we all understand the cost of that, which gets us into the whole conversation of shifting left, integrating code early and often. Uh, and, and, and ultimately C I C D, uh, in, in my experience and opinion, it is about building that, uh, uh, building that developer experience, uh, making it easy to commit code.

It shouldn't be a chore, right? Uh, committing code that's compliant, that's of high quality and say, meet security standards and requirements for your organization should be something that your development teams don't have to worry about. It should just be, should just happen.

It should be standard and it should curate, or, or, or, or a DevOps group And team should be able to curate workflows that allow that, uh, that developer experience to be absolutely phenomenal. I'm taking it. And to, uh, go ahead Jackie.

Yeah, And to kind of zoom out a little bit, when we start thinking about why organizations and, uh, and teams of developers are interested in adopting ci, it is to reduce toil in their teams and to drive efficiency and r o i of their processes. So when we start to think about what makes ci, CD and DevOps compelling, CI is a natural starting place because it's a, it's almost the maturest part of that integration, uh, system for organizations to then consume and generate stability. Although it can be argued that there's many different forms of CI that people start with, and when we compare it to modern CI CD capabilities, there's different ways of implementing that for, uh, legacy code and even large Monte repos.

But I would say that CI is the natural starting place for people because of the drive and the, the business factors, the logic of pushing them toward automating processes. And Ravi, you were gonna say something? I'd love to hear that.

Yeah, it, it's like kind of an approach I'll take. It's like I'll take a platform Turing approach, like there's an internal versus external customer type of approach. And so if we look at, I'll portray a platform engineer, like if we're looking at internal customers, it would be the development team.

What is an action that they do all through the day as a software engineer, uh, if you were to look, uh, at how software's written, it's not like, and everything kind of gets generated, it's trial and error. Like, if you were to watch me write software, you would think I'm stupid. Like, oh, I try things like a hundred times, uh, be before it worked.

And core to that is every time make modification, we have to make a build, right? And like constantly, constantly building artifact and local machines. So like to the panelists point, uh, get the kind of the first step is saying, oh, you know what?

We need to automate or in repeatable fashion what the developer's doing on their machine. We need to put it somewhere else so we can recreate at any point. And also have the ability to get an artifact and then deploy it.

So like, that's why a lot of automation starts with developers are already doing it time and time out. They know the build, they, you know, write the build scripts or the build manifestor recipes. Uh, and then packaging might be a different person who does that, for example, if there's a release manager or they have standards for how they release, um, but that's it, right?

Like constantly multiple developer tasks that are going on. That's where it starts to get automated in an external system. And that's why CI is a very valid place to start, um, in DevOps training.

And Adam raised an excellent point as well. It's more than just doing a build. There are things like code standards, security standards, there's all these different elements that you can begin adding to it.

And I mean, the first step is, does my code and Mitch's code play nice together? Let's, let's just figure that out. But is our code secure?

Is our code well written? Does it have, uh, is it documented? Well, these are all different code review processes, security analysis.

Is it highly performance? Is it, uh, there's all these different aspects of what our application needs to be at the end of the day that we can begin including as part of this continuous integration process. There's all these checks and balances that can be put in place to automate all the tedium that developers have to go through.

Some may even call those tests Some tests. Um, so guys, let me, let me, let me jump in here. You know, I was always taught if you don't learn history, you're destined to repeat it.

And so I think a little history here of ci, 'cause CI existed before there was cd. I think we lose sight of that sometimes. We think it's one, one thing with the slash in the middle CI cd, but it's not.

It was one existed before the other. com 2013, someone once said to me, DevOps, I forgot if DevOps is the disease or the symptom, but C I C D is the cure, right? And that, and that was really at the heart of what DevOps is.

And you know, CI certainly came first. We mentioned Jenkins, right? Jenkins is kind of the granddaddy of ci.

It wasn't really designed at first for cd, it was, it was really a CI product. They've, you know, I guess sometimes after we attach the slash cd, well, there, there are those who say that too. I, you know, another show we do is with the C D F for instance, the Continuous Delivery Foundation.

And I think later this month you'll see a report from them coming out on, on Jenkins use still for something that wasn't designed for it. There's a lot of people still using it for that, right? But there are other solutions.

There are more modern solutions. And part of the reason is, is because we've advanced the state of C I C D, right? CI began to introduce this automation because we were integrating code from more than one person.

If you're just gonna have a one person coding team, it's not much of a team. It's a coder. Um, but as we moved to teams, you, you started getting this integration as we moved to building stuff into getting it ready for deployment, such as testing and reviews and so forth, right?

com, the, the, the, the deployment wasn't automated. Someone had to physically push the button that said, deploy today's modern systems. You don't necessarily have to do that.

But it gave rise at some point to this idea of a C I C D pipeline. And to me, that was one of the seminal developments here. 'cause now we, we started thinking like a factory.

Ravi, you mentioned how code is done today. Well, the fact of the matter is, the coder doesn't write all the code today. The coder assembles a lot of components and then glues 'em together and makes sure they work or whatever.

But there's a p I calls and there's third party dependencies and everything else that go into this coded pipeline, right? And where we do that, hey, let's not wait to test that right before we deploy, let's move it into Git, right? Git ops, GitLab, the kinds of, you know, where, where these things are taking place.

We've seen a real evolution from just vanilla CI with Jenkins 10 years ago, 12 years ago, right? Harness and Robbie, I'm not, you know, asking you to blow your horn or anything, but Harness was one of those companies that started looking, let's take a new fresh approach to C I C D GitLab as well. Right?

Now everybody talks C I C D and pipelines and software supply chains and all that good stuff. You know, Alan, I think you're, you're, one of the things you're pointing out is that we turned our attention to the process of creating software. Yeah.

Not methodologies like the old days, you know, whatever. We still call it SS D L C. But, um, it, I think one of the things that also solved is, I remember, uh, integration Fridays, you know, you broke the build.

You don't go home until the build works, right? So we can test over the weekend. Many developers are happy for ci.

That's all, you know, that freed up a lot of our lives from having to stay the weekends and fix, you know, what that we did fix during the week. But my point is, and that's what it's allowed us to look more holistically at the, the pipeline, the, the development process. And think about it not as one workflow, but lots of workflows all happening, going through this process at the same time to help.

And that's part of what helps. And to me, That's the biggest innovation here, is thinking of this as a pipeline, as a factory. Yep.

With multiple different parallel executions going on. That's the continuous part of continuous integration. Good point.

It's often overlooked. It's often when you see it represented on a PowerPoint, you're like, okay, so then Mitch does this, then Brian does this, then Alan does this. No, Mitch and Brian might be doing the same thing at the same time.

And both pipelines are executing in parallel, and you have these multiple different concurrent operations all taking place, and it gets progressively more exciting the more stuff you add into that pipeline. Well, I think there's a couple of different things that happened, um, historically, right? Um, you know, when we begin to think about, uh, when we begin to think about business shifts over time, which really drove C I C D, right?

It was, it was, um, uh, things like globalization, right? How do I, how do I forget about the fact that we're in the same room? What if we're in different countries?

How, how does, how does that work? Right? So how do I, how do I work with somebody that's 5,000 or, you know, 5,000 miles away on another continent and begin to begin to integrate code from there?

Um, uh, uh, productivity increases from our, from our business needs. Especially when we think about mobility, right? When we think about when, when mobile, uh, uh, when we start.

So first started to having, having, uh, the internet came out, and then we started having as a service, and we had started having mobility that put a lot of pressure on companies to say, Hey, we gotta deliver faster. We gotta deliver with higher quality. We gotta deliver with, you know, security, blah, blah, blah, blah, blah.

All of a sudden that pushed back everything back down to engineering saying, well, I can't wait for my release engineer to go fix this problem or do these mergers. I have to know how to do those things, right? I have to know, I have to have that insight.

I have to be able to manage my pipelines. I have to be able to merge my own code. I have to be able to have systems that can help drive that automation for me.

And I can't wait for somebody else to go do that. And, and that's where we really began to drive that whole notion of, uh, well increasing more automation from a CI perspective and, uh, and a CD perspective. And that, that was coming out from increased pressure from our business, from our society.

It changed the culture. It totally shifted the culture of how we have to work as, as engineers and, and engineers have to know more and be able to do more, uh, beyond just write some code. Kind of piggybacking, oh, go ahead, Alan.

No, no, you go rah. Yeah. Piggybacking off of what Alan said, like if we take a look at like a modern C I C D pipeline, like what it is, it's a culmination of aggregation of expertise across the entire organization.

Like if we go back, you know, before Jenkins, before Hudson, before Oracle had the project, like it was a job or workflow runner, right? Like, I, I can't tell you the number of times that as an engineer, I did not know how to go to production. It's like, it was a mystery to me.

It's like, okay, I, you know, I wrote my more ear. I, I do not know who has to deploy this. Like, do I make a ticket?

Do I, you know, sit, do I go complain to my manager? Like, where's it go? It was a big disconnect because there was multiple people who were involved in the process.

You might have someone who had to test it, someone who had to release it, someone had to validate it today, if you like, with, uh, the codification of lots of infrastructure. So the rise of infrastructures code, uh, even, you know, the great equalizer, Kubernetes, like, we can write a YAML and expect, you know, parody between what's in one environment to the other. With, with the entire stack that Kubernetes provides, uh, the expertise is actually upfront, right?

We can codify, Hey, I know exactly how my application's being built. I know exactly where it's going based on certain manifest or recipes or configurations. And so yeah, it's like that culmination of all the expertise in the pipeline or pipelines is that it reduces that toil.

Like, I still remember, like even just thinking like 15 years ago, like, I'm, I don't actually remember how things got into production until like, oh, someone had a ticket, someone call somebody. But yeah, that's, yep. Well, Ravi, I definitely know how things got to production.

I pushed a button and I got there as a really engineer, as someone that was responsible for, uh, like hydrating Jenkins pipelines and pushing them into, you know, cloud native networks. It is, it's interesting to see the evolution of what we're thinking about, like platform engineering and this push for continuous, uh, deployment rather than just delivery. And I think that what's, what we're seeing in, in organizations that are slow to adapt to new technologies is that they're still relying on these old tools, um, because that's how CI was, was determined and defacto defined, um, you know, 20 years ago.

So I think what's happening now in the modernization of CI c d is that we're moving away from tools and it becoming like a practice and a team ethic. And that's why we're starting to see some like new texture and new interest and new ways of like, tools adapting to those, to those team processes. Like Ravi, you mentioned platform engineering, and I do think like Harness definitely celebrates the platform engineer, but there's still lots of organizations that haven't moved that way and teams that don't know how to work with platform engineers, I don't think they understand what the platform engineer role is.

And if you don't understand what the role is, how the heck can you hire someone for it? And it's a, it's a fascinating problem because it seems so obvious when it's described. You have your developers, they've been asked to implement new features, but they've only been given the feature definition.

So they start working and they build out this new feature, and then they give this code over into the release team to deploy into production. And all of that code was written on the latest version of SQL Server, but production's running SQL Server 2017. And now you have all these deployment challenges and nothing really works.

Why, why did we ever let the developers write on the latest version of sq? Why did we give them a platform and engineer this so that it was easy for them to instantiate all of this and they were building to the same secured platform that their code's gonna run in, in a production environment. And if they can't get it to work in Dev, I don't know what to tell you.

We gotta figure that out before we go any further down this process. Uh, that's in my mind exactly what platform engineering is supposed to be providing. I totally agree.

And, and that's where we have IDPs, um, development platforms, and that's where we begin to curate workflow. And that's where we, we move away from, uh, providing generic services to developers to go do whatever they want to providing platforms to curate workflows so they can complete or accomplish exactly what you laid out, Brian. Yeah.

And then, and then it provides guidance for the ops team. It's one of the great truisms, right? It's DevOps, dev and ops.

It's not dev dev, dev, dev, dev, dev, dev, dev, dev, dev, dev, dev and ops. And the ops team has a vested interest in managing that platform pipeline as well, so that when you're planning a production upgrade of your database, your web servers, whatever else, you feed that into the development pipeline so that they begin working on the code so that that code will be ready at the same time you've done your upgrades and you synchronize all these different things together. It's really, really impactful when you start doing that.

And it's amazing to me how few people actually do. Yep. We're going back, you know, the benefit of hindsight, I think one of the mistakes that we've had made in the DevOps movement is frankly, we've put too much on the developer.

We have, you know, the developer is kind of the, uh, alpha predator in the food chain. And, and, and the, and for many of us, the answer in DevOps was, yeah, no, the developer will do that. Security being more secure, do more security testing, take more security.

We gotta make the developer security champion, uh, platforms. Yeah, no, the developer wants to pick what platform it is. They're really smart.

We pay them a lot of money. They want to decide what platform they wanna work on, otherwise they're not gonna work here. And we, we gotta keep 'em at all costs testing, let's get rid of the test engineer that we'll, we'll make automated continuous testing that the developer kicks off, Right?

And, and it's Fascinating to me because everybody talks about shift left, and so many people talk about it in the context of make the developer do it. Yeah. They're busy.

That's What they have a job already. That's not what shift left. And you know what exactly.

Developers like to develop, they like to code. They don't necessarily wanna pick that platform. They don't wanna do testing, they don't wanna do, they Don't want, they certainly don't wanna do testing.

They don't Wanna do necessarily security either. They don't wanna write in secure code, but they, they're not security professionals, right? We, we've seen this movement where DevOps is DevSecOps, right?

I I learned that phrase, Jackie from, uh, who's the, who's your C C M O, Ashley, Ashley, er, Ashley Kramer, Right? Ashley taught me that DevOps is DevSecOps. That that is exactly my position as well.

I actually hate the term DevSecOps because I would say to people, what kind of lunatics are you that you run a DevOps tool chain that doesn't include security? What are you, what are doing? That's, it seems insane to me to do it that way.

And I get that people like the term and they wanna differentiate. I I do not. I'm like, no, it should always include security.

I, And I love your, I love, oh, Go ahead. I just like conversations that start out with who are the lunatics? And I'm saying, Jack, I'm lunatic.

I'm not lunatic. Because I think that there's value in being explicit about it. Because if we think about historically, what people were doing, if they were managing upstream dependencies, they were checking vulnerabilities after compilation, and after the product was built at the final release package, meaning all the vulnerabilities were detected right before production, rather than it being explicit in the process from the beginning and doing security checks at every step of the way.

I mean, can you imagine like getting all of your CVEs flagged two days before you're supposed to trigger deployment? I'm, I'm, I'm in the performance engineering space. Of course, I can imagine that everybody waits till the last minute, right?

Load test. So I, so I would say I'm, I'm a big fan of saying DevSecOps because there has been an illusion that you can have DevOps without, without security. And that's why there's plenty of people running without dedicated security tools or even security tools inside of their CI I C D tool chain.

Yeah. It's a dissemination of expertise again, right? So Like, if you look at any organization and you look at harness like, you know, we have hundreds of developers, but we have a handful of AppSec engineers or, or we have a handful of sre.

And so that, that shifting left mentality, like I'm actually very pessimistic about the shifting left. 'cause there's only so much, like there's full stack developer, or like, you write it, you run it type of model, it's like extremely stressful. Uh, it's all about paying off tech debt early and avoiding it.

So it's, it's like, hey, for, to uh, Jackie's point, if you can make more hygienic decisions, um, as you're building or as you're writing and creating something, you would typically make the more hygienic decision. It's just that you don't know what you don't know, right? And so this is why, again, the, the pipeline is like this big disseminator of, oh, you know what?

Our AppSec engineers have blessed these methodologies that are there. You can clearly see what you're subject to before you go to production. Even, you know, some of the stuff we have i d e level integrations with, it's like you can get that feedback very, very quickly.

And, and you don't have to pinging an apps like engineer in Slack, like, Hey, can you look at this? 'cause again, that will get us into the old model of you wait right till you deploy, you know, you're, you're find some stuff out that maybe not be too pretty for you. Yeah, wait, waiting till right before deployment is, uh, that's hope as a strategy.

That's all that is. You're just hoping that your security scan is gonna work out well, or that you're gonna run a performance test and we're just validating it's gonna be great, right? Um, but there's, there's no time in your project timeline to deal with.

What if it's not, it's the, the, the classic, well, I don't know what, I don't know. I don't know the waiting till the right moments that the, the proper place for that. And it ties into a lot of things.

And, uh, Ravi, you brought it up again, right? Shift left doesn't mean make the developers do it. I do a lot of talks to QA teams and a QA organization got very lazy in the early two thousands.

They were looking at their test management platform, whatever that happened to be, and they were viewing that as the center of the quality universe. And when DevOps started to rise right around 20 10, 20 12, that switched to the CI system. It wasn't the test management platform that decided what tests needed to be run.

It was the CI system that triggered those tests now, and it became the center of that. And QA was slow to adapt. This explains the rise of JMeter.

This explains the rise of Selenium. This is the developers going, I can't wait for the QA teams to keep up with me and my release cadence. I gotta do it myself.

And Shift left is still tied to that perception. I talked to the QA teams. I'm like, you're gonna have to learn how to write workflow code in a CI system.

You're gonna have to learn how to use a source code management repository like Git. You need to check your code into those Git repos. You have to work with the tooling the developers use.

And I get it, it's hard. It's new, it's foreign, it's weird, but it's not hard. The developers know exactly what they're doing and how these tools work.

Talk to them. They're not your enemy. You're all working towards a common goal.

And I don't understand the breakdown in communication, uh, between these different organizations. Brian, I'm interested to hear your perspective on this. So something that we at GitLab do, we, we do a report on a tool chain sprawl, and we ask people and developers like, what's the toil or the time spent, uh, for your developers maintaining and integrating their DevOps tools?

So do you think that there's more toil spent and the developers maintaining and integrating tools or working across their functional partners? Because I think that's where, like Shift Left is supposed to be trying to solve both worlds. But I'm curious Daycare, which, which one you see?

Yeah, it's, uh, it's both. Every customer is their own special unique set of, uh, fascinating processes and technology decisions and integration challenges and political structures that, you know, never, you can't talk to them because I don't like the director of that group sort of nonsense that I see, uh, occasionally as well. These types of challenges, I guess it's, uh, it's incumbent on the organization to do that return on investment.

Look at it. Is it worth the time that I'm spending maintaining these integrations and investing in this for what I'm getting out of that effort? Right?

You don't automate for automation's sake. You automate because it's solving a problem and fixing something, increasing your velocity. That's usually the number one goal.

Uh, there is that old adage in the QA space, you can have an on time on budget, bug free pick two. Um, our DevOps offers a real opportunity to let you have all three. And it's just by, as the code is written, the moment you have something that is in any sort of a testable state, you begin running tests against it and do that quality control effort.

This is also tied into the great myth of QA organizations. That QA teams are responsible for quality assurance. This is false, uh, every which way from Sunday, everybody's responsible for qa, QA teams are responsible for quality control.

There is a big difference between those things. And it diminishes the value of all the tests the developers do every unit test, they run all of those Selenium JMeter assets. They might be building the asserts, they're putting in their code.

This is all quality effort and it can all be captured by the CI platform and become useful information for a quality organization. If I look at two different streams where two different dev teams are building two different APIs, and this dev team has 500 unit tests covering 95% of the lines of code in that project, and this team has zero, where should I prioritize my time as a QA team? It shouldn't be a 50 50 split between those two things.

I'm gonna have to spend much more time on the completely untested a p i that I'm receiving through this pipeline. But I also think this is where we get into, and sorry, I'll go back to our IDPs again, right? Because this is where we begin to shift into building out those curated workflows and standards across, across our pipeline.

So there, it, it is not necessarily, uh, developers who want to integrate some tool for shifting left. It's, it's more a here's what the development organization needs. We're going to use these tools.

We'll do the integration as platform engineers. You don't have to worry about that. You worry about the output, right?

So for example, a a a a qua, whether it's a quality issue or a security issue, and or, and I'm, when I talk about engineers, I'm, I'm grouping them all together, whether they're qa, security, whomever, right? You guys worry about that. We'll worry about curating the workflows and make sure we're working in the same way, although we may be delivering different things.

So we have to know a, a lot about our, our products, what they need, but we also have to be able to enable our developers so that they can deliver right? Their code, integrate their code, deliver their code in a consistent way. So, you know, we don't say, Hey, which, which, uh, you know, what's your favorite, uh, source control system?

We're, we're gonna support that. It's, it's the opposite. We say, Hey guys, uh, we're gonna use this.

This is the source control system for the entire organization. We're gonna use this tool for critical vulnerability management. This is how it's gonna be integrated.

This is your role in, in that integration. This is our role. This is how this works.

And so we go from, we step away from, Hey, here's a bunch of services guys. You go ahead and go use them and build whatever you want to. No, this is how we work.

This is the roles, here are the people. This is what we need to do in order to achieve that business outcome. Exactly.

Here's the technology and the process you're gonna follow to leverage that technology and the expected outcomes that you're going to get from using that technology. And don't get me wrong, developers may or engineers may come back again when I use the, oh yeah, a holistic thing may come back and say, well, but that doesn't, that's does, that's not realistic. That doesn't work.

Okay, let's talk about that. Let's, let's, let's accomplish that. And we, and it is not, it's not meant to be a, uh, uh, uh, uh, a we or, or a or a, uh, like a fiat, right?

It's a US conversation, right? Yeah. Not, not us versus them kind of thing.

That, that Goes against basic, right? It's never us versus them with DevOps guys, we're, we're, we're running a little low on time. There's one more aspect of this whole C I C D building block thing I want to introduce.

And that is scale, right? 'cause that's, that's what really put the, the zinger into things here, right? When we were releasing software once a year, twice a year, maybe once a quarter, right?

All of these things we've spoken about were relatively manageable. You didn't even need to automate too much. 'cause you had so much, well, if you spoke to the developers and the teams, you didn't have enough time, but you had a lot of time, right?

One of the, you know, building blocks of the whole DevOps movement was, I, I think it's John Spars, uh, presentation, you know, 10 x or 10 releases a day, something like that, right? And, and that's what we've come to, right? Where it used to be fantasy to think we were gonna release 10 x a day.

But many, many, many organizations are doing that today. But even the organizations who did once a year are now releasing once a month, that's a 12 x jump from where they were. That kind of scale of release demands, automation, and has driven, I think a lot of the evolution into C I C D pipelines, right?

Looking back at that, if you ever get a chance, I think it's on YouTube, go back and check out all spa's presentation from that back then, and you, and you see how much of that is actually driving what, where we are today. Thoughts on scale. Okay, Jackie, I was saying, and arguably we're starting to see the conversation shift from like the continuous testing, the continuous security to continuous compliance and creating gates for each step of the way.

6 times more likely to deploy multiple times a day. And it's because they have like the trust and safety in their system. Hmm.

And that goes back to your scale point, because when you have C I C D configured both deployment and the CI continuous integration piece configured in a way where there's, there's gates, there's approvals, there's proof points, there's like a balance of automation and intervention. Uh, people are more likely to allow individual teams to self-control and self-monitor and self-release. And that, that's really empowering for your developers, but also for platform engineering teams if they are supporting these individual developers at scale.

But I'm interested to hear what Robbie says. I feel like he might have a different perspective coming from the harness world. Yeah.

It's all about confidence, right? Like it boils onto to that, like, are you confident, uh, with the decisions that have been made? Because like a, a deployment is an aggregation and culmination of all the decisions that have been made from the previous release to the current release where you are.

And, and so everything that we talked about, how do we, how do we have more testing? They're all, how do we have more resiliency? They're all confidence building exercises.

So a lot of it's also psychological or, or organizational design, right? Like people have gotten burned before, so they have over, you know, they, they have innovation birth control, they, they lock down control versus they stifle innovation a little bit. And so the, the greatness about a C I C D pipeline is that it's like a recipe.

It's a recipe or a wiki, right? All the decisions that were made are visible in this pipeline. So to Jackie's point, or to Adam's point or Brian's point, like, Hey, here are the test results.

Go look at it here. Here's, you know, this is where it deployed. This is the soap testing that we did.

Go look, the results are attached to the OR link from the pipeline, so you can take a look at it. And with that confidence building, you can deploy a lot quicker. But for me, it's organizational design that usually stops people from deploying as much as they want.

Yeah. And, and I was gonna say, and Ravi and, and Jackie, I, I agree with you and, and Alan, I, I agree with you too. I mean, I think it comes back to having solid metrics that and, and clear objective measurements that, you know, everybody agrees on, uh, that says, okay, okay, everybody, here's what we wanna, here are the metrics we want to see, and this is what defines release, right?

We hit these thresholds, we're good to go. And then you work backwards, right? And you automate those, you automate those, uh, decision making processes, right?

It is no different than building sort of an activity diagram, right? It did it hit, did we hit this metric? No.

Okay, we're not good to release. Yes. Go to the next step and we can automate those, that decision logic.

And you know, at the end of the day, uh, be able to say at least objectively, this is good to go. Right? And, you know, maybe, maybe at some point in time we'll add a little, we'll add a little intelligence there that says, well, historically we've seen, we're good to go, but you know what, we're really not good to go because we found these other issues that, you know, we found in the field after the fact.

So how do we, how do we go ahead and maybe add some extra testing to and adjust our test cycle to, to, you know, look for these particular things, right? But it, but to me, at the end, it is solid metrics, working backwards, uh, uh, uh, adjusting your, your pipeline, your automation to make sure that you're hitting those metrics and then releasing with confidence. I mean, I love it because you're talking about test driven development.

Exactly. How do You get your outcome? Uh, I'm a huge quality proponent.

It's been my whole career, and it's even more than that. It's just making the results visible. Who are the people who are interested in looking at this and how are they gonna consume that information?

I can't give unit test result reports to the senior executives, nor can I give a high level executive report to the individual contributor developers that need to have the right information presented in the right context in the platforms and systems that they currently use to execute their day-to-day job function. It does mean no good to give a performance engineering platform capability to a dev organization if they have to log into it to look at results or do anything with it. They need to be able to get the data presented in the pipeline or attached into their I D E or wherever else.

Right? Agreed. Agreed.

Guys, we're, we're, uh, we're really getting to the nitty gritty end here. I wanted to give each of you like a chance to just leave each of the four of you a chance to leave with our audience. You know, your final thoughts on this whole notion of C I C D and C I C D pipelines as building blocks, which is the theme of our, our series right now.

Um, we'll go in reverse order. I think, Adam, I introduced you less. So we'll let you go first and then we'll do Jackie, then Ravi, then Brian, and then Mitch, if you don't mind wrapping it up.

We'll, we'll go to that. Adam, go ahead. Sure.

Thanks Alan. So I'll make it quick. I mean, I think it's all about curation of workflows through, through IDPs, right?

I, I think we're making, or we're beginning to see a major shift in the industry from providing generic services that help facilitate generic C I C D to more curated workflows through, uh, internal developer platform. So I think building an internal development platform and go going with platform engineering is, is sort of the future of, um, where our industry is headed. Fair enough.

I think when in deciding that you need to modernize your C I C D architecture, really think about the business value and metrics you're trying to join, like deliver upfront is that will help you decide on the tools you're looking for that will help you decide on how much your developer's going to do, how much headcount you need, rather than just relying on, uh, a tool chain that you currently have. It gets very mindful of you to like step back and look at your entire system and your entire entire development org and look at the math that you're trying to deliver for your developers and for your business. Kind of, uh, closing, closing glos here.

Like treat your C I C D pipeline as code, pretty much all modern solutions, your pipelines as code. Don't be afraid to iterate over the pipeline. Your pipelines will change over time as new methodologies come out, new paradigms come out, new technologies come out, they will adjust.

And having that mindset of set and forget, it's not acceptable. You'll be refining, uh, or building and iterating over your pipelines as your software engineers are doing that with the code base that they work on. And so, yeah, don't be afraid to change and just, you know, treat it as a piece of software.

And my final thoughts are that continuous integration, hugely valuable, continuous deployment, hugely valuable. And both of those form the underlying, I guess, infrastructure requirement to embark on a continuous quality journey and getting, getting the automated testing in place to ensure that every time somebody checks in code, you validate that your system is still functioning the way that you want it to, that it's still performant and that it's still secure. You Know, Alan, I think this conversation really speaks to CI or CD isn't a discreet step.

It's part of a broader workflow. It's part of a pipeline, parallel pipelines, et cetera. A software development process.

Stephen Covey has the quote, begin with the end in mind, have some perspective about where you're gonna take this. You don't have to be an expert at all of it, but come listen to these experts. These are the folks can share with you.

You know, there's a lot of things we touched on today, so I, I think I'm gonna rerun this recording right when we get done 'cause there's some great ideas I didn't have time to write down. So listen to this again, if you're listening to it for the first time, There's a lot in here for sure. Thanks, Mitchell.

Thanks. Thanks, Adam. Thanks Brian.

Thanks Jackie. Thanks Ravi. Thank you for watching this today.

Many, many special thanks to our friends at t Tricentis for partnering with us on the DevOps Unbound, uh, video series of which this is part of. Um, I hope, I hope you enjoyed this as much as I think all of us enjoyed doing it. So until next time, this is Alan Shimel for Techstrong at DevOps Unbound.

You've just seen another episode. We'll see you again soon.