DevOps, ITIL and AI: A Recipe for Better Results? – SKILup Days 2024

com, tech Strong, ISM what, uh, Textron AI Security Boulevard, text on tv, a whole bunch of Textron stuff. But I'm here today moderating a Greek panel for you all on Skill Up Day. We're, we're happy you're joining us.

This month's Skill Up Day is a pretty, pretty cool one. It's dealing, of course, with DevOps and the relationship of DevOps and Idol and how to be successful with it. I've got a fantastic panel of folks that I want to introduce you to who's gonna discuss this topic with me, and let me introduce them to you.

I'm gonna start from, well, on my screen, he, he's on the bottom. I'm gonna go from the bottom up. He's someone I've had the pleasure of knowing, I don't know, 12 plus years through a variety of different roles.

Uh, one of the smartest people I knew on in C-I-H-C-D for a long time, my friend Eric Minnick. Hey, Eric, if you wouldn't mind introducing yourself. Hey, thanks, Alan.

Uh, yeah, as you're saying, I've been in the DevOps space for a long time, uh, previously with Urban Code and IBM, and now I'm really enjoying life as a director of DevOps Solutions at Harness. Uh, co-author of application release and Deployment for Dummies. And very excited to be here with you today.

Thank you. Thank you, Eric. It's great to have you on.

Um, next up I want to introduce you to Paul Davis. Paul, I'm gonna let you kind of fill the audience in on your background. So, hi.

Yeah, my name's Paul Davis. I am Jay Foxfield ciso, and you might be saying CISO DevOps ile. Why?

Well, Uh, I had the honor of, uh, being exposed to ILE many, many years ago, and it's really 'cause security people are process Wes. We like consistency, and we also like to have a common vernacular, and ILE works well on that and making sure we have a common language to talk through. And its evolution over the past few years has been fairly amazing as it's modernized itself to reflect the sort of dev development world.

So, um, I'm happy to be here, um, looking forward to the conversation, so thank you. Well, I'm looking forward to hearing your views on this. Um, I've, I've got some questions for you, but first I need to introduce our last, but certainly not least, panel member today.

It's my friend, hope Lynch. Hope. Why don't you tell folks a little about yourself?

Certainly. Thank you, Alan. Uh, hope Lynch, senior director of platform at CloudBees, uh, started my career in it.

So very familiar with ITIL and the importance that it ascribed to it, uh, transitioned, uh, over time towards more engineering strategy work. Um, but definitely, uh, see the, the intersection there. Uh, spent time at Cisco, spent time at Red Hat now here at CloudBees and enjoying it.

Thank you. Welcome. Thanks for being here.

Hope so. Let, let's kick right into it, guys and gals. com, uh, 10, 11 years ago, Eric, as he mentioned, was an urban code and idol and, and, and, uh, DevOps were not exactly peanut butter and chocolate.

Paul, I think you hit on one of the main reasons right off the bat, which is I is all about standardization, all about sort of, uh, you know, launching in lockstep and following very specific, uh, protocols and policies and processes. DevOps, when it first came on the scene, as, you know, the new kid on the block, you know, refused to be defined it well. There was no manifesto, there was no best practices.

There was. They, they actually relished in that they kind of, in many ways flew from the seat of their pants reacting and iterating based upon feedback loops. And, and so a lot of the early DevOps people just looked at idle as everything that was wrong with it and develop it.

And what DevOps was set out to fix the idle folks. As you said, Paul relish the idea of, of of, of knowing what comes next, of knowing what, you know, following a process. Everyone's on board, everyone knows what's you know's going on, and it just, it just seemed, you know, that they were banging ads and they didn't seem a way for these two frameworks to work together.

Um, what changed, Paul, I'm gonna let you kick it off. You, you mentioned it in your opening remarks. What do you think has changed that?

Or maybe it hasn't changed? What Do you think? Oh, it has changed.

Um, development's moving a lot faster now. You know, we are agile as you meet the manifesto and all that sort of stuff. You talked about that.

But the other thing is, is that software is coming far more dependent on the IT infrastructure. And so the speed that we need to move at and the speed we need to deploy means that we have to have a way of communicating across the teams to say, this is what I need to do, this needs to get in production, this needs to happen, et cetera. And so that whole thing is sort of driving us through.

And now when I gonna put my dev hat on with the regulations, wanting consistent secure software supply chains, mm-hmm, those two, it's almost like they were developed in two separate parts of the world, but they all have the same vision. It's the whole, the same steps there. We might have different terminology for the stages, but in the end, what everybody wants to see, whether you are a customer of a software company or you are delivering software, you're a business, is that consistent process from beginning to end.

And that's what I love about sort of version four is it does pivot that cross and create that connection. So for me, I think it's a sort of a, Hey, we are actually talking about the same thing. We, we might wanna use different words, but we have the same goals.

And I think that's a big thing there that we want to drive through. Oh, Barry, yeah, I, uh, I agree and I think there are some factors, um, other factors driving this convergence as well, right? So everyone talks about digital transformation, that is, that is a big job and DevOps teams are engaged there, but those transformations are often led and managed by it.

Um, that structured approach that ITIL can bring can be a great compliment to the way, uh, DevOps looking at more rapid, uh, iterative cycles as, as you had mentioned. So aligning those two. Um, looking at how incident and problem management, uh, works with the CICD pipelines, how change and release can work, uh, with automated deployments.

Um, how we can use IEC for configuration management. There are a lot of ways these things can go together. Uh, not that it's easy, um, but, but it is complimentary.

Yeah, I, I agree. And I, it may be a happy accident, uh, but I, I think ITIL has for a long time had the structure in it to support a DevOps approach. And that if you have a standard change, uh, hopefully I'm, I'm getting standard to normal, right?

But if you, if you have a standard change, right? That's something that's low risk, there's a defined procedure for it, and because it is low risk and standard, you don't need to go through a cab. You can just do it, right?

And what has DevOps been trying to do this whole time is make software delivery a well-defined process. It's the pipeline, right? You push a button or you just commit code, it happens.

You have a standard process and through all of the testing, through all of the security scans, through all the safety nets you have watching production, you're driving down risk, right? The risk that it's gonna fail the blast radius when it does fail. And so you drive down the risk, you have standard procedure, now you're a normal change.

You don't have to go to the cab. And so where I think we instinctively looked at this and said, oh, those ITIL guys want us to go to the cab anytime we do a deployment. Yeah.

So that's incompatible with DevOps. No, the, the gate was always there. The bar was always there.

Make it low enough risk, you don't have to go to the cab. Yeah. Socially did that always work?

No, but I think we're getting there socially. No. Yeah.

So to me there was a little bit of, you know, Muhammad coming to the mountain, but the mountain came to Muhammad a little too. Yeah, right. I, I think Paul, you mentioned ILE four now ILE four is no longer new.

ILE four has been now probably three, four years now. Right? But it was a watershed moment for IO and, and it's wrap approachment or Reproachment with DevOps and Agile and these kinds of, you know, frameworks.

I think it really was a game changer. It it's, it was a little bit of, you can't beat 'em, join them type of, of a moment. But at the same time, I think DevOps matured a little bit.

It wasn't just, it was no longer the wise kid, wise ass kid hanging out in the school yard. DevOps teams realized it's a big world out there in the IT real, and they need to work with everyone. They can't afford to alienate teams of people like this, right?

And, and that DevOps in and of itself in a vacuum would not be successful. They needed to work with the ops teams and with the ITSM folks. And, and we see it even now with platform engineering, SRE, all of these folks.

It, it literally, as someone once wrote, it takes a village to develop and deploy and manage software to manage it. And, you know, I think that DevOps world has, has recognized that most of us and and so no longer looks at it as some, you know, archaic dinosaur or something that we're here to replace. We're not here to replace them.

Um, you know, so it, it was, you know, both sides of, of it coming together. I now hope I I'll give it to you with CloudBees, right? Yeah.

You guys have a, a ton of Jenkins users have in the past of, of Legacy. Um, do you got, have you seen that? I mean, is that accurate to what you would see it at CloudBees, you think?

Yes. Um, and like you've said, um, as before, is no longer new, but some people who don't know about the changes in itil, right, are still thinking about the heavy process oriented framework or, you know, that service lifecycle model versus the look to something that is more adaptive and more value driven. And one of the areas that does show up is having conversations with customers, um, about value stream management, right?

Once you start talking about value stream, those, those processes come in, those connections with the development teams come in. And even if they aren't using the words itil, right? You, you have to use some of those processes if you, if you want to smooth your path.

So, so we do see that, we do hear it when we're talking to customers. And though some of them may not realize it, they are, they are on that road to convergence Fair. Paul, Eric, thoughts on that?

I, I think that the, so I, I had the glamorous, uh, journey of dealing with iel, the first version. Um, at that time I was running a global security program and they said, Hey Paul, we are moving the whole organization to IL and you need security to move to IL. And I went, okay.

And it ended up running two iterations of IL one where I talked to the rest of the world and then one for the infrastructure that my security team was running, right? Um, and I bring, and I, when I look at the, the sort of the iterations of ILE over time, I still feel that there is a gap. And I'm kind of conflicted 'cause I don't wanna corrupt ILE for what it's doing, but I think there needs some mapping to be done between you, you know, you scared me when you say get cabs.

That really scared me. Um, I must admit that I, I, I big changes security paranoia, but from the perspective of like getting the security gates in place at the right points, automating it and stuff like that, and aligning that with, um, you know, the idle framework and things like that, I think is important. Um, but it's almost like the, the, I, I don't wanna say this, but there needs to be another iteration, or maybe I'm biased 'cause I'm a security person, but I, I feel that there needs to be some synergy.

I mean, 'cause problem and incident, the number of times I've had to explain this is an incident now has become a problem is like, okay, yeah, instant's over really, but tidy up well, how the incident lasted a week, but the tidy up takes six months a problem. Like e So that kind of thing. What worries me, um, from that perspective, um, and I think there's, I dunno what the others think about this idea of alignment Vocabulary is so hard.

Incident problem, I was tripping over standard and normal. Um, you know, deploying release, uh, you, we, we have synonyms that mean very different things all over the place. Um, so I, I don't know if additional iterations fix that or if, if that's just the challenge of, uh, having a lot of similar ideas in our world and maybe we can't use individual words to describe things.

Speaking of that, I think one way to get over this hump, um, you know, me, sometimes I'm an optimist. Sometimes I tell Alan I have a controversial opinion. This may be one.

But, um, instead words numbers, right? So there are KPIs that could be meaningful, um, across, right. There could be a dashboard where it is, how long does it take us to recover that will be meaningful to the idle team that will be meaning to full, uh, to the DevOps team.

Uh, something that, you know, hopefully is customer centric metrics, um, that can give some real time monitoring some long-term trends. And I think that helps you to a degree get over some of the issues with language and just have the numbers and the trends that the teams can look at that they are both, uh, impacted. Yeah.

And, and the more customer-centric, the more business-centric those numbers are. Yes. The easier it is to get everyone on the same side, right?

Yes. You know, we wanna be making more money, we gotta deliver some innovation. Um, we're not gonna be successful.

We're not gonna serve our customers if the site is slow or down or leaking their information. Um, but those sorts of very clear business aligned measurements Mm-Hmm. Get people on the same side.

And that helps. But it's kind of funny that you, we, I mean, business KPI is really important because it, it's, it's like the translation layer between what we do Mm-Hmm. And what the business understands we do.

And there are huge disconnects in that space. It's just horrible to think about. But the thing for me is, is when you talk to the developers, the feeder of this content to DevOps, they don't think about uptime availability, you know, continuous improvement.

They're told, Hey, write this application, get it out quickly. And their metrics don't always roll through. So I sometimes worry that I doesn't translate all the way down the chain.

You know what I mean? Sure, sure. Go ahead, Pope.

I'm sorry. Did, I didn't. Oh, that's okay.

And, and I think, um, you know, there are levels, uh, in DevOps as there are across the organization, right? So maybe you ask someone who's the manager of the team who owns translating that into something that is meaningful goals and OKRs for the team to hit. They may never look, look up the ladder as far, um, to understand a lot of those business metrics.

But some of these things should be reflected in, in some of the OKRs and other things. I think, uh, that the team's trying Agreed. Go ahead.

How I was gonna say, how about this for an OKR? Um, do you know 25% of a developer's time is spent fixing bugs and vulnerabilities? Yes.

Yep. Yep. And, and here's the horrible thing is the executives are suddenly realizing, Hey, we've got this, this impacts our availability.

'cause it's a risk, not availability, not the system's running. We have less change windows 'cause we don't have to keep upgrading and patching things all over the place. They'll be really nice sort of thing.

Um, so the, the whole thing of sort of the, the metrics, um, I think it's a key thing. I know CISOs we're stressing a lot now 'cause we have to think about supply chains and software, and we have to now still start talking to new teams about this stuff. But it, it worries me that, that those metrics, those KPIs, um, they have to translate.

And I'm, I'm counseling a number of CISOs now on stop talking about vulnerabilities and talk about business impact. And this does have me thinking maybe I need to be brainwashing them with idle. Yes.

Sounds like a good plan to take over the World. But, but you know, seriously, let me interject in there a sec. I think when I, I mentioned some of the things that were driving ITIL and DevOps to come closer together.

We, I i, I omitted security and, and Paul, you reminded me, right? When you have today Jfr, for instance, right? The emphasis on DevSecOps and security and compliance, it's the same for CloudBees, it's the same for harness.

In fact, it's the same for every DevOps company. GitLab, GitHub, you know, name a DevOps company. They call themselves DevSecOps companies.

Yep. Right? And, and quite frankly, security and it's, and, and how it interacts in an ILE organization.

ILE has become a friend in many ways to the security team. So this is another sort of gravity, well, if you will, that are pulling these two planets, you know, orbits closer together and, and, and making sense why we need to work together, why we need to work together in there. And, and we shouldn't, you know, I I think we have to acknowledge the role that security plays across the entire factory, if you will.

So, I, I, I hate the factory analogy, but, um, but, but I, I, I would say that very, very short term, you can be sloppy and bad and fast, but the only way over long term to be fast and still be in business is to drive exceptionally high quality. And that's testing and it's all your security stuff, right? Those things go together.

They're hand in hand. Um, at least conceptually, right? Security problems are bugs.

Um, performance issues are bugs. All of these are bugs. And to drive that quality is what allows you to have rapid change.

It's what gets trust in what you are doing. That's the gate is quality and security is arguably the most important part of that right now. Agreed.

You, you've all fallen into the trap. Remember I said we need to align ILE with, with security. You, you brainwashed you all, you see reverse logic.

I said teach the CISOs Isle. You now say, yeah, but, but you, you're right. I, I mean, the thing for it is really we are, we have this terrible thing called the triad, which is the confidentiality, integrity and availability.

And it's a curse because it means, um, we can't say no to protecting the organization. And, and DevOps is part of that journey, you know? Agreed.

That is. I, um, guys, we have less than 10 minutes left. I, I, I left this topic to the end.

I, I hope it doesn't run away with us, but I, I wanted to, you gotta mention AI right there. 'cause everything's AI today and, and the role it's playing, whether real or not, it, it certainly is sucking a lot of oxygen in the room. Where, where do you see AI playing in this relation, this budding leather affair, if you will, of DevOps and itel and security?

Where is it today and where do you think it might be in the near term future? Who, who would like to take first crack at that one? We're all jumping in.

Okay. Sorry. Go for it.

Hold, go on. Come on, come on. I was gonna say that, um, when, when we bring in ai, we talk about the, the one that, that comes to mind for everyone, and they think about generative ai.

And if we also think back to what you were saying about quality, right? Uh, there is an opportunity to relieve one of the biggest pain points when I was in it. If there's an incident, getting good documentation, getting timely documentation, getting all the information, what an opportunity to use AI to assist with that documentation, perhaps through some observability tools, perhaps through, um, you know, conversations that have been tracked in Slack, et cetera, right?

And bringing that information together. So now you have someone who needs to review. It's been documented and hopefully, um, it's easy to search even if it's documented in the past, I have had to search through, uh, confluence documents other things to find a past incident to reference, right?

For a current incident. Um, the, the slowness and the amount of time that takes, I, I think if I were still doing that work now, I would be chomping at the bit, uh, for the changes, uh, that AI can help with. I, I was talking to the, our director, VP of, uh, cloud engineering yesterday, and he was talking, when we have incidents, we're using AI to yes.

Find those past issues to get remediation. I like that's, that's today. Yes.

Um, and I mean, AI's everywhere and it, gen AI is a little new, but other aspects have been around for a while. Like in the testing tools, um, the screen scrappers have been AI powered for a long time to automate building out tests. Uh, monitoring side has been rich in machine learning and AI for a long time.

It helps you reduce the risk of a deployment, right? You know, that you're doing a deployment, you apply some machine learning across your observability suites, you go, oh, doesn't look good. You roll it back before things are on fire, right?

Helping detect smoke before fire. Um, but I think a lot of these things have been fairly optional for a long time. But as developers are becoming more productive using tools like copilot mm-Hmm, everything downstream needs to be so much better.

So much crisper to keep up with an increased pace of innovation. Um, AI is widespread now. It's gonna be everywhere and it's gonna be critical as we go forward.

I, I, I'm gonna be slightly controversial. Okay, good. Go ahead.

Okay. So AI is in its early stages. Um, first of all, please do not use chat GPT to generate your incident report.

Okay? True. This is build your own Yeah.

Career altering event if you do that without reading it and checking it. Okay. Yes.

So I, I, I work, you know, I've been working with AI for many years from as you talked about it. And I think, I think we need to like things like one of the most exciting things I heard recently is we could use it to generate documentation. Ho ho it looks like a lovely thing to do.

If any you could do the release notes, that would be brilliant as well. But I think that's pushing it a bit far. But the thing for me is when we look at ai, the great thing you can do, if you know how to do what I call socially engineering chat, GPT, Is Get it to tell you about exceptions.

Don't get it to create, but say, what am I missing? And if you learn how to use these AI agents, um, and you know how to put the guardrails in place and give it very clear instructions about what to do, then it can help you. This is another tool in our tool set.

And as we were discussing, I think it's like, Hey, I will solve everything. No, but it will accelerate things, which means for DevOps teams, it's gonna get, there's gonna be more, more things to look at. More volume, more.

So automation is key. But I do like the, you know, as you talk, hope about the using it to find previous instance data, but also be very careful about who has access to that data. 'cause you can have PII in there and you gained a whole new game.

Let's not even talk credit cards because, so you have to be very careful about what you expose to those in those data sets. Control it. So treat it cautiously.

Don't trust it, but use it as another tool in your tool set. 'cause it's gonna make life, life a lot easier. Mm-Hmm.

And that's really a key thing though. That last sentence, it's another tool in the tool set. It's not, it's not Skynet, it's not magic, it's not, you know, any, it's a tool and tools, you know, tools make the person, um, I was thinking though, like Paul, you mentioned, and we all have seen the figures, right?

Developers spend something like 11, 18% of their time, uh, fixing bugs, closing vulnerabilities, patching, if you will, or, you know, remediating vulnerabilities. That to me is the kind of stuff that we, we can maybe see AI take a better role going forward, right? Because we're already using AI to find vulnerabilities, you know?

Right. And, and seeing if they're viable. But can we not only find them, but fix them in one fell swoop and free up 10% more of a developer's time?

What's 10% of a developer's time works, Right? I, yeah, I, I I can believe the thing for me is, is it's all, like everything we always talk about shift left and everything, the, the quicker we can spine the problem and solve the problem, the less work it causes further down the pipeline. And the key thing is, is enabling the developers to understand and give them tools that they can use properly to discover a bug, an issue or a problem, or a weak, you know, they haven't cleaned their input.

Another stat top in the top five, we still have buffer overflow, cross site scripting and not cleaning your inputting. I mean, it is just like secrets. I mean, all these things, if we could get a web and say, hold on a second before you admit this, uh, you've got a bug here, you can fix this.

And, and we can, and using AI to do that passing and you know, looking in front of both of the point of view of the source code, the source code quality, but also the quality of the external libraries and open source components using here is a key thing. If we can block it there, then we can show improved KPIs, da da da, reduce the load on DevOps, make sure that we can handle the increased volume because there's more, there's software develops, not slowing down, it's accelerating. And that means we've gotta find ways of dealing with it.

So, yeah, So, So I'm thinking maybe you have found a great unifier, right? Between ITIL and, and DevOps. It, you know, developers don't want anything that's gonna increase their workload or they feel waste their time.

ITIL teams are, are trying to meet the demands for compliance and governance and oversight. Um, but if there's some AI magic, call it magic for now in the middle, right? That can alleviate the workload but improve, uh, the traceability of some of those processes.

And yeah, I mean we, we have the code, we have the scanners that are telling us that there's a CDE here. Um, you know, the tooling is increasingly getting to the point where it'll put two and two together, use some LLM and say, look, it's this line of code. Let me explain the CVE to you in plain English and a code change that's gonna resolve the CVE looks something like this, right?

Um, and whether we want it to actually just do the pull request to work That, That's a little more debatable, A little nervous there, but At minimum, um, I don't think we're very far from it being very difficult to put in the full request. Yeah. And the trick is gonna be just like on your incident report, just like if you're using chat GPT to do your homework or anything else, like you gotta read it and you gotta own that output at the end of the day and make that change, right?

You can get off the blank page, you can get guidance on where to go with some generated code or whatever. Um, but at the end of the day, it's your name on the, the commit. Uh, you're the one to approving it.

You're the one on the hook for it. And we need people to do this just like, you know, we have autopilot, but we still have some humans in the cockpit for when things go wrong and to keep an eye on the autopilot. Alright, um, guys, unfortunately we're at our 30 minute time that we were supposed to go here or we're over that, but it's okay.

Uh, Paul Davis, JFR Hope Lynch, CloudBees, Eric Minnick harness. Thank you all. Thank all three of you for joining me today on this panel.

I hope our audience found it interesting. Look, I will say that IT and DevOps have become chocolate and peanut butter with a little security on top. So, and AI maybe, you know, maybe the, the missing secret ingredient as well.

I hope you've enjoyed this. Uh, there's plenty more today on our skill updates for DevOps and Idol. So please stick around for other sessions.

But until next time, this is a Shimel for Techstrong. Have a great day everyone.