Bridging the Gap: Integrating Security into CI/CD for Seamless DevSecOps – The CD Pipeline EP17

Hey everyone, it's Alan Shimel here from Techron Group, and you're watching another episode of the CD Pipeline. The CD Pipeline is a, uh, partnership between the CD Foundation of the Lennox Foundation, and here us here at Techstrong. We're about, about once a month we try to bring you some of the latest topics regarding or germane to the CDF audience, to the CD Foundation.

For those of you who're not familiar with the CD Foundation, we'll usually have someone here from the CD Foundation. Um, but for those not familiar, the CD Foundation, as I mentioned, is a daughter foundation of the Linux Foundation, but it's responsible actually for the management upkeep running of several of the largest tools in the CD universe, CI/CD universe, including Jenkins, uh, Spinnaker, um, Tracy, one of yours, Orillia Orillia. Um, I think there's eight or nine different pro programs that fall under the auspices of the CDF, but it's more than just managing those, it's, it's working groups around the security of them, of operating them best practices.

It is the community for CI/CD. And so this show, CD Pipeline tries to capture that. Um, this week's show is challenges and wins in integrating security tooling into CI/CD workflows.

Let me introduce you to our panel for today, and then we can jump right into the topic. First of all, I think it's our first time on, and if I am mistaken, right, Kate, it is your first time. My First time.

Very cool. And I'm gonna try not to mess up her name, but I forget things for one second to the next, uh, Kate Scar, Uh, Kate Scarcella Scarcella. If my eyes were better, I'd be able to read it up there because my, they didn't put it on my prompt here for me like they're supposed to, but they try.

All right, Kate, beyond Scarcella, what else can we learn, learn about you here today? So I am a cybersecurity architect. Uh, I got my Master's of Science and Information Security and did my thesis on securing the electrical grid in North America.

And I graduated way back in 2006. And I tell people this because there was only four people in my graduating class, so nobody was really talking about cybersecurity, even though it was coming up in, um, in some products. You know, you had av, you had networking security, and dare I mention the start of Identity Nexus management, which was like, ah, but anyway, so that's my background, and I did it for Very cool.

Yeah, so very, For long, that was the electrical grid without Texas. Very good. So, so, um, I'm not gonna touch that one again, but I, I've been in security about 25 years, myself more now.

And, and you're right. Back then it was, it was network security or endpoint secure host security as we called it. And, uh, it certainly has changed over the years, though.

It's become more important than ever, obviously. Are you still working in the security field today? I am.

I took a short sabbatical for about a year, and I'm just, uh, coming back out of that sabbatical, um, connected with Tracy, who has, um, who has, you know, full speed here this year. So, yes. Um, Well, it's great to have you on here.

Thank you. Thank you. I'm, I'm very happy.

Appreciate it. Looking forward to, to hearing more. Um, next up we've got Ryan Ware, as in software.

Brian, welcome to CD Pipeline. Tell us about yourself. Hey, thanks, Alan.

Uh, I'm really happy to be here my first time as well. Uh, uh, I, I appreciate Tracy Ragan me, uh, uh, to come on here. Um, so, uh, I've been doing security in one way or another for 27 years, uh, or so.

Uh, I'm a software developer by, by nature. But, uh, uh, you know, early in my career, uh, I first started at Intel doing, uh, implementing security features into digital rights management stacks. Uh, later I did offensive security research into, uh, Intel's products.

Uh, later I was more of a security architect and then, uh, worked in, uh, the realm of open source for quite some time, uh, on one. Uh, how do, uh, uh, product teams incorporate open source in a secure way, as well as how do you securely, uh, uh, make and contribute to open source externally? Uh, Intel has a, a, a a, a large presence in the open source community, uh, these days.

Uh, I work for Carrier, where I am Deputy Chief Product Security Officer, uh, focusing on security tooling, uh, CI/CD training, uh, secure development practices, uh, and, uh, I'm also in the, the Open Source Security Foundation where I am the chair for the security tooling working group in that organization. Very cool. You mentioned Intel's commitment to open source.

I think it's very fashionable today to, to crap on Intel, right? They're not, they're not Nvidia and you know, how the mightier fall, but I, I think people have never given Intel enough credit for their commitment to open source, open source communities, open standards, and, you know, kudos to them for the work they do. I know they work very closely with the Linux Foundation, C-N-C-F-O-S-S-F, you know, a lot of the, the foundation.

So Absolutely to that. They do, they do an amazing job with that and still do even, uh, with all the troubles they're having right now. com/intel, so, Absolutely.

So shout out to them. All right. Our last panel member today is our friend Tracy Ragan.

Tracy, of course, CEO of Deploy hub, uh, Aurelius, one of the products under the CD foundation came out of Deploy Hub and, and their, their efforts. Tracy's also, I mentioned she's a host on our Techron gang, and, you know, she works, she has a hand in a lot of different Linux Foundation, open source, uh, projects and, and boards, including OSSF, open Source Security, OSSF, and the, uh, DD Foundation among others. Tracy, great to have you on today.

Well, thank you. Yes, I am, um, a busy girl. I kind of have one foot in the security world and one foot in the DevOps world.

I am on the board of the Technology Oversight Committee at the CD Foundation. I've, uh, served in that role now for, uh, several years. And I'm also on the board of the open SSF, um, so I'm keeping kind of my thumb, um, on the heartbeat of both sides.

And we re at the CDF, we recently started a, um, special interest group called CIC CD Cybersecurity, which, which Kate is the chairperson of. Uh, and it, the reason we, I felt we needed to start it was because there isn't a strong enough handshake between what the security side of the business is doing and what the dev, what the DevOps side of the business is doing. So it's time that we have that, uh, conversation.

It's a really critical one. Um, most DevOps engineers, you know, even just putting in, uh, the scanning of A SOM is a major undertaking, and there's so much more to do. So it makes me a little nervous that we are so far behind the eight ball.

Uh, I wanna remind everybody that it takes us about a hundred days to respond to a vulnerability. It takes a attacker less 10 days to exploit it. So we are, um, at a major disadvantage here, and there has to be a discussion on how to improve that time.

And so we're hoping that we can do that with this, uh, the c the new SIG at this, uh, CD foundation. So I'll use CDs out there, which I call you, please. CDs, I want you to join the, uh, the CI/CD cybersecurity sig and start, he helping us with really defining what that looks like in the pipeline.

And I hope we can have a deeper conversation around that, considering we have two security experts on the call today. Absolutely. So, as I mentioned, I've been in security 25 plus years, right?

com was I felt that DevOps and the whole CI/CD pipeline model was a way to get like a second bite at the apple for security. We could correct a lot of past wrongs by moving further left, shifting left into the development pipeline to fix security problems that by the, by the time we saw them pop up in production, they were a lot harder to fix. It would be much easier to fix them further left.

It sounded great, right? com devs, the whole rise of DevSecOps, as we call it, right? Uh, we made a lot of progress over those 10, 11, 12 years, but most recently there's been a pushback where have we shifted left too far?

And by that, have we put too much of the onus on security on the developer who's not a security person, right? And, but nevertheless, you know, we, we've made them the, the focal point for our security efforts and, you know, pre-deployment security where instead of, let's say, maybe building it holistically into the whole pipeline process right from left to right. And, and so, you know, there's been pushback.

Hey, instead of shift left, we should shift everywhere. We, we, we need to build security into testing. We need it built into the pipe, not onto the developer's shoulders.

Now, now, Kate, you've got your master's degree in, in all of these things, and, and you're, you're the head of the sig. What can we do to it? It can't just be, let's make the developer our security or make the dev our security person.

He's, he or she is not. What, what can we do? What are we doing?

What should we do, you think, in, in terms of integrating security into this workflow? Well, I think, and when you, um, go out, uh, to the specific page that, that we have, one of the areas, security is very complex, as you know. Um, cybersecurity is very complex, and it has becomes, the complexity in itself has, is also a vulnerability.

So we need to make things simple. So yes, are we putting too much on the developer and should we have it throughout the pipeline? Absolutely.

But we also need to have, um, bite size, you know, these, these, you know, Lunchable type of, of packages that we can say we're gonna do, you know, we're gonna secure in the, in the, you know, free deployment phase and how, what does this look like? And I think this, the more simple that we have the tools, because as, you know, to introduce a new tool, um, is, is just a headache for our developers, and yet another tool and another tool and another tool. And so I think we need to have, um, tools that are, and they're very expensive.

So tools that are, you know, open source that can be used, that can be used easily, and so that it doesn't take a master's degree to go out to try to figure out, well, how am I gonna secure this? I think that's, you know, something that we have talked about, um, with the sig. Um, and just, it's so important to keep it simple.

I mean, it, it sounds, you know, we ridiculous, but we have made it so hard. We have made cybersecurity so difficult to consume and so expensive. I mean, think about the tools that we have developed.

I worked with IBM for over 20 years, and it's not just, you know, it wasn't just one tool. It wasn't just, you know, um, static, you know, analysis, coding, it became, I mean, you just, everything just grew and grew and grew, and we just would keep adding and adding. So I think we need to do more with less.

So, you know, the one tool can take on this pipeline from left and throughout. I dunno, Tracy and Ryan, what, what do you guys think? Well, I, you know, I, I preach simplicity all the time.

Um, part of the problem with the CD pipeline, if I just put my DevOps hat on and not my security side, um, the problem with the, the CD pipeline is it's so brittle. Um, everything's based on, uh, a script. So we have to go update all those scripts.

And that is not an easy task, folks. It is really not easy to manually update so many thousands of scripts, thousands and thousands of workflows. Um, so it becomes a challenge because we don't wanna touch those workflows.

They break easily. So then maybe we have to have a security workflow that the, that our workflow calls. So, Kate, as you pointed out, we've made everything so complex.

Everything not only just not security's complex, but so is our workflows. They're complex too. So we've dug ourselves in a bit of a hole, and we're behind the eight ball.

So how do we get out of it? Now, many people know from my discussions that I have been a big fan of CD events. Let's rebuild and redefine how this, the, the pipeline works.

But that e, even though IBM has done a great job, and that some of the team on that project has done a great job of defining what those events look like. And even, uh, Jenkins has an event, has a CD events plugin. We, I don't see, uh, the, what I like to call the giants, the Microsofts, and the, um, the, the intel, even though they've been doing a great job in some areas, I don't see them understanding why events are important, why it's important to re uh, to disrupt how we do pipelines.

So as long as we have this, uh, this difficulty updating pipelines, I think we'll have difficulty implementing tools. So what we have to be able to do is define the low hanging fruit. Let's at least get started with getting us, uh, a software bill of material generated and make that a, a, a common mantra that we can really expose to the DevOps pipeline and the DevOps engineers to say, this is one way we can get started.

At least let's start there. So, simplicity, I think, will be critical. Yeah, I, I agree with that, Tracy.

Um, I, I would like to go back, uh, uh, uh, to what Alan was saying for a minute though, and, and just push back slightly on the idea that developers need to be security experts. I, I don't, I don't think developers need to be security experts, but developers do need to be capable of writing quality code. If, if, if we don't think that there's an expectation on them to, to write quality code, uh, um, then I, I, I think there's something fundamentally broken in, in the system as, uh, they're the ones that are writing the code.

Uh, and security in a lot of ways is, uh, you know, many, many security vulnerabilities are just engineering 1 0 1 quality issues, uh, buffer, overflows, uh, uh, uh, null point or de references, things like that. Um, that said, I, I, I do, uh, uh, like what you were just saying, Tracy, uh, about events, I, I, I think one of the, the problems with how we have incorporated tooling into, uh, pipelines is, you know, it's all about, okay, how do we get this tool in here and get results out of it? And, and that's not the focus it should be on, it should be on what's the activity that the developer is doing right now, and what's the information that we can get from our tools that would be helpful for the developer to have during this activity?

A great example is pull requests with, uh, on gi. Uh, for example, uh, uh, you know, a lot of, a lot of organizations use, uh, uh, static application security testing tools, uh, traditionally called static code analysis. Um, which by the way, uh, there's open source tools that have been making great strides in this area.

GCC fourteens, uh, static analyzer, uh, functionality is, is way improved over previous versions. Um, that's said, you know, the right time to be able to show a developer about flaws in their coal code using a SaaS tool is during poll request time. And, and ensuring that, that, you know, when a developer needs the information about the quality and security of their code, it's important for 'em to have it at the right time.

Uh, traditionally we've told developers, Hey, yeah, you have to go over to this other place over here where, where, uh, the results for, for all the scans are stored. Developers hate doing that. They won't do it, and we're never gonna win by share doing it that way.

So let me weigh in here. I bet you if we did a survey of developers and said, how many of you want to develop low quality code? Not a lot of them are raising their heads.

Every software developer I've ever met just about has a tremendous amount of pride in, in what they do and the code they develop. You know, what used to be, before we got into the age of DevOps and pipelines and the software factory that we have today, software development was very much sort of like a, a guild, right? Ancient, uh, not ancient, but you know, like old German guilds where there was a lot of pride in craftsmanship and stuff like that.

When I was a security guy, I used to think, boy, those people don't give a hoot about security. And if they did, we'd be better off. But then when I, the more I got into DevOps, the more I learned that they do give a hoot about security and quality, right?

Because security is synonymous with quality, and they do give a hoot. And then early on in DevSecOps, a lot of security companies said, you give a who to bet security, Mr. And Mrs.

Developer, here's a tool to use, use our tool, use our tool, use our tool. And you know what? And that's where it went off the rails.

You, they don't, they're not gonna, they use a security tool. When you start telling a developer, Hey, wait a second, we wanna do a static analysis scan of your code, and that's not enough. Hold on.

I wanna do a dynamic scan analysis of your code. Wait, there's more. I see you've been using a lot of that open source stuff.

We're gonna do an SCA software composition analysis scan of your code and whatever the, and I just bought this latest company's greatest new, you know, ICAS, so whatever the heck they're calling the next one. And the developer, they just, Hey, can you just tell me if there's bugs in my coat so I could fix it? That's all they wanna know A hundred percent.

Right? And, and we've, I think we lose sight of that. And, and in a perfect world, that's where, in along this pipeline, these things get done and it makes its way back there, right?

And the code gets fixed. And look, here's the good news. We live in a wild time right now with AI and automation and everything.

A lot of these things could be fixed on the fly like that, right? I mean, you know, stuff we dreamed about Yeah. In 2006, right?

The, the ability to do automated remediation. I, I was selling vulnerability management in 2006, you know, no one wanted it, it, it took 90 to 120 days to remediate code that was code in production, not code in, in, in de development. So, okay, have we Gotten any better at that?

And still that? No. We, we have, because I'm gonna tell you, the problem I had back then is we were able to identify vulnerabilities and we built workflow into our product that pointed to the patch, and we had the ability, 'cause we also developed a NAC product network access control.

We had the ability to remediate on the fly. No one would let us, yeah, yes, no one would let us because they were afraid to patch without first testing to make sure that it didn't break something else. It might break your stuff, Right?

We can't, I'd rather be insecure than have broken stuff. I don't necessarily agree with that logic, but nevertheless, that was the prevailing logic. Have, have we changed?

Ryan? You've been around. We're trying.

Yeah, we're trying. But did that means we haven't changed? Is that what you're saying?

Yes. I, I would, I would say there, there are areas in the industry where, where it hasn't changed and change is hard. Uh, um, and, and to be honest, I I work in one of those industries right now.

Uh, um, one of the things that blew me away when I came to work for Carrier is the support life for some of our products. I, I mean, I, I used to work on automotive stuff where they're talking about support life of eight to 10 years. Uh, we have to support software stacks in our products for 25 to 30 years.

And, and because of that, the interesting legacy implications of some of the software we have, uh, uh, just are, are an interesting challenge for us in, in the new ecosystems. In, in a lot of people are resistant to change because of that. Agreed.

Right? You're talking about critical infrastructure. I mean, right, Brian?

I mean, you know, You, you know about it. Kate, Windows XP baby, you know, good luck. Um, you know, the one thing that I think would help us, and it's the antithesis of when we think about cybersecurity, but you know, the, the bad actors are doing this.

And that is, you know, when we talk about open source, you know, they actually, the, the bad actors actually do it, right? They collaborate, right? The reason they're able to get their stuff done so quickly is they have such huge collaboration tools.

And I actually think they're doing it right. I mean, it's a, you know, crazy idea. But I think we need the, you know, the, the, the, those who are trying to make things better that we really need to, to be, you know, have open source, you know, from, so from, you know, coming from all these, you know, companies that were, you know, doing proprietary stuff, you know, I've been like this new open source, you know, let's collaborate.

Let's, I think it's gonna be one of the most important things because when we have this discussion about, you know, the pipeline and can it just be on, on, you know, the, the, the shoulders of, of the very beginning person, you know, when we talk about cybersecurity, one of the things that we would talk about is, is everybody's, you know, everybody has to know about cybersecurity. Everybody has to understand it from the user with their, you know, digital user interface with their, um, mobile phone, uh, which has become a human machine interface to so much to, um, you know, to, to the worker, because we are one in the same, right? You know, that the, the home user is also the one who's going into the office, who's also working with a, you know, we all need to think about cybersecurity differently, um, in order to help things to become, I mean, it, it sounds, you know, somewhat esoteric, but it, we really, we need to, to do it.

So it's not so scary, right? Because we sell based on people being scared. And we need to, we need to really back up from that and think, what are we gonna do to make it better?

I think, and, you know, to, to Tracy's point about, you know, low hanging fruit, boy, how much can we, I I, how much can we do with just getting the low hanging fruit? You know? I mean that, I, I think I, I think it could be like a 70% type of thing.

I, and, you know, maybe that's a crazy percentile, but, you know, long hanging fruit, I mean, that's a, it's a great idea. There's definitive there, absolutely. But there's a whole bunch.

If you could just, you know, just take those, you know, we've got a few minutes left. Let me bring up another kind of push pull that I think really affects us. And that is, and this has been going on, bro, as long as I'm in security, which is what's my tolerance for security slowing things down?

Yeah. Well, right, that's a good Question. That's, that's not a little question, You know, because that same survey where I asked the developers, do you like making low qual, you know, crafting low quality clo uh, code?

The next question is, you know, why don't you do security better? And it's always because the pressure is on, and I get paid based upon how many lines of code I publish, and I don't have enough time to write and test. We don't have enough time to write and test the code thoroughly because we have deadlines to meet.

And so when security becomes the people who say no and puts the brake on going faster, we very quickly get kinda shoved out to the side, pushed to the back, you know, stay outta the way of this. You're, you're standing in the way of progress. Um, how do we, and, and again, this was one of the things about DevSecOps that I thought we would do better.

How do we overcome that perception and move at the speed of business? I like to say moving at the speed of DevOps. Okay, moving at the speed of DevOps.

So we have a weird, there's a, you know, there's a, the cultures between the DevOps people and the culture between the security people could not be more different. Um, if we think about, uh, a new vulnerability that's been found in production, the mindset of security is don't tell anybody. Go through a, uh, event management process.

Notify the people, only the people that should know. Because if we, uh, tell everybody and let the development team who's being impacted know that they may have, we may have some kind of internal attack to it. So there is a hold this, you know, hold the cards close to the chest mentality inside the inside on the security side.

It just is there. They don't, they lack the trust. They want the control of managing it.

This slows everything down. The, the, our, our culture there has, is, is just wrong. Now remember, it may have started back in the, in 2000, Alan, when you were trying to solve the problem.

And what they were doing is they were, you know, if a, a, a bug was found, uh, uh, if a hacker found a vulnerability that impacted Microsoft or IBM or the government, or Hewlett Packard or anybody else, they got, they were purchased. So we had a zero day market and nobody told anybody about them. So that's where we began our story in security.

Now, on the other side, we have DevOps engineers who have been preaching agile development and releasing fast for the last 10 years. And we have gotten really good at it. And on top of that, we have Kubernetes, which means everything's decoupled.

Everybody use, it builds their own container. So that one production vulnerability could be living in hundreds of containers across our endpoints. So now we find ourselves having to fix some really big problems with a culture that doesn't want anybody to know about it.

And another culture who says, you guys are way too slow. We need to continue pushing new innovation across the pipeline because that's what we're being told to do, and we're trying to build the best quality code we can. And we're trying to manage DevOps pipeline so that they're getting code out as quickly as possible.

Because that's what the business demands and security sitting there and saying, well, we don't want anybody to really know about this. Let us try to mitigate it and go through the process and only tell the teams that need to know that this problem happens. It doesn't work.

Those two cultures are, are conflicting. So somehow we have to create that handshake is kind of what I began this conversation. Security has to be more willing to open up the door and let more people know about it.

Security has to be willing to have those fixes pushed across. And Alan, you were just way before your time. In our world, what we're trying to do, you know, for Intelius and to play up, we or we, Orillia has the information right now to be able to push out a remediation.

So we have discussions about discovery and how to remediate. We wanna shift the focus to how does DevOps fix it? How can we use DevOps information to at least create, update a, a helm chart or a, a Docker file, and at least create a pull request for a high security vulnerability that's impacting certain teams and get it to them as quick as possible so they can make a decision if they want to move it forward, they can do the analysis and they can accept or reject that pull request.

But to do that, we have to get the security teams to say, yeah, that would work. That would be okay. But at the moment, they may not be saying that 'cause they're saying, no, we wanna manage that response ourselves, and it slows everything down and the world just keeps turning while security figures out what they need to do.

I, I think that's beautiful. You're right on. Yeah.

I, I mean, we, we in the cybersecurity, um, Alan and Ryan, I mean, right? I mean, we, it we need to change this mindset that we have, and I really do believe that we need to be more open about the vulnerabilities that we have, or we're not, we're not making it the way we were doing it. We know that.

So it's about time that we switch things up and say, okay, let's try something different. Yeah, I, I, I agree with, with what both Tracy and Kate were saying. Uh, I, I'd also just add to, uh, uh, taking a, a, a little bit of a, uh, uh, a perspective change.

I, I think security folks a lot of times lose sight of what developers really need, uh, to, to do their jobs right, and, and do their jobs the way security folks think they should do their jobs. Uh, uh, uh, and, and a lot of the things that, that security professionals ask teams to do without thinking about it, uh, uh, dramatically slow development down. 5 times the build time.

And building the Linux kernel is not a, a small event anyway. But you can't ask dev teams to, to wait for, for 2, 3, 4, 5 hours for results on tools until a pull request before they can go accepted. Uh, uh, you, you know, you, you have to figure out what's the right balance, uh, uh, to be able to, uh, uh, bring the bar up from where teams are doing it right now, to, to, uh, doing it in a way that, that they feel is acceptable use of, of their bandwidth and their time.

Agreed. Agreed. Hey guys, we're, we're about outta time here, unfortunately, you know, we didn't mention the CDF.

You mentioned the, uh, STIG that you guys started. org, isn't it? foundation, Excuse me, CD Foundation.

CD Foundation, yes. And the SIG is new. It just started in January.

And our first exercise is we're going through the, um, secure software development framework. We're looking at every single task that relates to the pipeline, and we're associating open source tools that can be used to, to accomplish the task. I love it.

Can you get to the sig off of CD Foundation page? Yes, you should be able to go to the, um, uh, community page and find it. Alrighty, Kate, congratulations on leading the SIG there and getting your hand into this open source security world.

It's going to, we, we can all use the help. So thank you for your, for your efforts there, Ryan, same to you, right? It sounds like you've been involved in this for a while now, whether it through Carrier or Intel, what have you.

And thank you for all you are doing. Thank you. I appreciate It all.

I just have one more, I just have one more thing before we sign off. I wanna shout out. Shout out to Sasha, uh, Wharton, who is one of our, uh, top or like Ortel contributors, and he was recently nominated to the, uh, the CDF uh, governing board as a open source representative.

So we're super proud of him. Congratulations to Sasha as well. Alrighty, that's it for CD pipelines.

We'll be back next month with more, but until then, you can get all of your CD information at CD Foundation. Until that is Alan Shimel for Techstrong. Thanks everyone.

Bye-bye.