Why Dev Teams Still Ship Vulnerable Code: Insights from Cypress Data Defense

Hey guys. Thanks Viro. We're here with Aaron Kura, who's uh, director of Cybersecurity for Cypress Stated Defense.

And they have a new report out on application security that among other things, proves that we are knowingly shipping insecure code into production environments. Aaron, welcome the show. Thanks.

Appreciate you having me. Walk us through the high points of this study that you guys did. What and most importantly, you know, besides the fact that we are shipping code that we know has issues, anything else in here that kinda left that at you or surprised you?

Well, it's interesting because we've known forever that people have problems with application security. It's one of those things that as security professionals, we see it all the time. We see that they don't have time to do security, they run through their sprints, they run through, Hey, we need these features.

The business says, Hey, we've gotta have these things to make money. Okay, we take, we wrote the code, we push the code in, we run through, and then we get to the end of the sprint and hey, we don't have any cycles to do. We'll do it next time.

And they bring it around and it goes and goes. And then they get it out to production. It's like, Hey, we found this problem.

We'll do it next time. And so they end up with this huge technical debt and no time to fix it. We end up with configuration issues.

We end up with all of these problems that just never get fixed. And then there's a breach and they're like, how did this happen? But what really showed up in this survey was that they know that it's happening.

They know that there are issues, they know that they have big security problems and they knowingly push it to prod. And that's really the biggest thing that came out and was surprising to me is that 62% of 'em said, yeah, we're pushing things out there with big security holes. Yeah.

So the road to hell is still paved with good intentions and as part of the good intentions that they think that, well, we'll just get to it in the next cycle. 'cause the next thing will be a a, a sprint is right around the corner and we'll just tuck it into that, but then it just falls off the table or what happens. Yeah, they, if we can just get to that one sprint where we can dedicate a whole sprint just to cleaning up those security issues and if we can just, we'll hire a security person and they can come in and they can do all of those things for us.

And then what, we don't have budget this time, but if we can add it into the budget next year and it just kind of falls down the list of importance as we just get enough to keep going and we just get enough to make it last. And it never becomes enough of a priority until that big hat comes and then it's about getting through the breach. And we can hire the forensics people, we can figure out what happened, we get through the breach and then okay, well we made it through that.

Well our stock is still selling. In fact, after a breach those stocks always go up. So let's just try to scrape by a couple more years.

And it just keeps going. The cycle perpetuates. I think though, that things are about to change.

'cause if I'm not mistaken, I think I've seen studies in the past that say only a small percentage of vulnerabilities are actually exploited. However, in the age of ai, it seems like the bad guys are gonna be able to A, scan the code and B, ask the LLM to help them figure out how to exploit a vulnerability that the LL LM is gonna help them find. So is this whole thing about to ratchet up into maybe, uh, you know, the Pied Pipers coming home, the roost as it were, It's gonna be a really interesting thing to watch because not only are the bad guys gonna be able to look at the LMS and say, okay, go look at this website.

In fact, there's a ton of tools coming out for us pentesters to send it at a website, let it go, figure out all the vulnerabilities for it. While obviously the bad guys use the same tools that we do, but all of the developers are using those same LLMs like copilot and things like that to write the code. So we're gonna have AI, fighting AI to go back and forth to have this huge war over who's gonna write and kill the most secure code.

It's gonna be fun to watch. I'm really excited about it. It seems like though the first wave of that is a bunch LMS that were trained on bad examples of code that they pulled from across the internet.

So are things likely to get worse before they get better? You are not kidding. A lot of it was trained on open source code that some of it was just really bad.

One of my favorite things to do is to take a piece of code that I've written and ask the LLM to make it better and it'll come back and it'll take and suggest some improvements. And I look at it and I say, is this really better? And invariably it comes back, oh, you're right.

This doesn't even compile. Let me try again. And after four or five iterations, it'll come up with some interesting improvements, but I don't know that they're even better.

And so to take and put all of your faith into an LLM and say, can you write this for me? And just let it come up with something and say, yeah, that's probably good enough, and push it forward. We're not at that point yet.

We have to have people watching those and knowing what good code looks like and what it's coming out of. So I agree with you that we are not at that point where we can trust that code to be useful. And I think the code that's coming out of them is arguably worse than we were getting just out of plain brand new developers that were coming out.

So, And oddly enough, truth being stranger than fiction, but apparently, um, the LMS are pointing to software packages that don't exist and the bad guys are getting wind of that and creating those software packages, sticking 'em in repositories and loading 'em up with malware. So, um, have we kind of like, you know, just come full circle on chaos. I've seen a lot of that.

I've seen those same stories and I've seen those packages and they're terrible. It's just like, hey, here's a great opportunity. If it's gonna suggest it, why not capitalize on it and stick 'em in there?

And then we've got some other supply chain issues where we're having people spend two years to become trusted in these spaces so they can then put their malware into these packages to become part of this ecosphere that Hey, I am a trusted person so I can put my garbage package in there and push throughout the entire place. It amazes me that now we have to take these things that have been a close knit group and say, okay, do we need to be closer? And how do we verify all of these things and these, these packages that are just showing up that, yeah, well the LLM invented it, so let's take advantage.

Mm-hmm. Um, we talk a lot about DevSecOps lately and people are investing and I like to think we're making some level of progress there. But, uh, there's still some debate about who should be responsible for application security.

Is it totally in the hands of the dev team and or do cybersecurity folks have a role to play and how do we get them all to play nicely together? I like that you said DevSecOps because it's very rarely said that way. It's mostly DevOps and then all of US security folks are like, Hey, you keep forgetting to invite us.

We want to be in there. So the whole DevSecOps part of it is that security should be everywhere. It should be as part of the development cycle.

It should be part of the pre-planning, Hey, how are we going to include security in this? And it needs to be part of the CICD process. It needs to be part of the deployment.

How are we gonna check for it afterwards? Are we doing scanning of the production stuff? Those things that have been in there for five years, as those new patches are being placed to the oss, what is that doing to those apps that have been sitting there?

So it needs to be part of all of those groups. It needs to be part of development, it needs to be part of operations, it needs to be part of the whole cycle. How do we do that?

'cause some folks will say, you know, we have hundreds of developers and two security people and there's not enough of them to go around. So how do we kind of get some sort of knowledge sharing going between a very small number of people and a lot of people who probably can't even spell cybersecurity. Yeah.

We get ask that all the time. I teach for Sands and we do a lot of people that come in and say, Hey, I am the security team. We have 150 developers.

How do I do this? Say, well, one of the things you can do is outsource a lot of your security stuff. Go to a managed security provider, managed AppSec provider, let them run your scans, let them do your static analysis.

Let them do your, uh, you know, your SASS and your das so that you can spend time building a security champion program so that on your dev teams, you have one person on each team that has some interest in security, you can meet with them once a month, buy 'em some pizza and say, Hey, let's talk about this vulnerability. Or how do we mitigate these kinds of things? Or what kind of things are you running into on your team so that you can at least let some of that trickle down and get some support for those two people who are just running ragged right now.

Mm-hmm. Um, we invested in a lot of tools over the last few years and related to securing our software supply chains, but it always seems to me we're kind of like trying to shift everything so far left and yet maybe the focal point of the development process is the repository itself where all the code is. So maybe that's where we should be thinking about applying our security time and effort.

Or is it really need to be at the IDE level where the developer, you know, unless the thing is automated, isn't gonna really pay attention to a million alerts he doesn't understand. Yeah, I think there's a little of all of that. So I think you need to have some static analysis going on every time that code is checked in as well as the integration into the IDE so that when they throw something in there, it pops up and says, Hey, this is SQL injection.

You don't wanna do concatenated sql, this is a great place to use, uh, an ORM or this is a great place to use something else that is not this. So having that feedback to the developer in a language that they can actually understand versus, Hey, here's this esoteric thing that came out of a scanner that you have no idea how to understand is huge. Um, one of the things that I have always found a little perplexing is it, it doesn't feel to me like there's a whole lot of accountability for these vulnerabilities when something goes on.

And it seems like maybe that's because it's perceived as to be a team error or we just, you know, so desperate for developers that we don't wanna call anybody out too much in case they're gonna leave. But is there something that I can get to that feels like more accountability without necessarily having to, you know, put everybody on trial every time there's a problem? Absolutely.

That's one of the things that actually popped up in our application survey was that there are a lot of developers that are afraid that if a vulnerability is discovered or a breach happens that they're gonna get fired. And that is such a terrible place to be in that you're afraid to write code. When I switched over from being a developer to being in the security team, I was afraid to write code for like six months because everything I wrote was wrong.

Oh, that could get hacked. Oh, that could get hacked because I knew better. Why didn't I as a developer know better?

We should have done a better job educating developers, teaching them how to write better code, teaching them how to write more secure code. So I think having a better education program to being able to talk to those devs, having more brown bags, that security champion program that, hey, here's a cool article I found about 'cause just like every other profession, we want to be good at our jobs. We want to do better.

We want to actually be proud of our craft. And as a developer, I wanted to write good code, I wanted to write clean code, I wanted to write fast code, I wanted to write efficient code, and I wanted to write secure code. And just having somebody that I could ask, is this right?

Having somebody look over my shoulder and say, Ooh, that thing that you just did there, you shouldn't do it that way, is huge. Where are there regulators in this conversation? 'cause I feel like, well in certain countries at least their restrictions are getting more, uh, rigorous and in other places not so much.

Um, so are we making progress there? And what is the right level of regulation? That's a really good question.

I kinda go back and forth. The biggest issue I have with regulation is that finding the right size, the right place, the right application. If we look at something like PCI, applying PCI to a mom and pop versus applying PCI to a multi-billion dollar corporation, they're different things.

And to rightsize that and to put it into a place that can be applied to each of them equally and the right way is hard. And then to abstract that to a place where a government can apply it and then to regulate it, it becomes a very difficult conversation. So I'm not sure that there is a right way to look at that and say, this is how we need to do it from a regulatory standpoint.

There have been some states that have kind of taken that on and, and done it more from a, we're not saying how you should do it, but if you don't protect the data that you are entrusted to protect, then you're gonna be held liable. So that I think has some merit because we don't care how you do it, but if you don't do it, there's a problem. So I think that may be the right way to go.

So what is that one kind of maybe simple thing that you wish more people were doing that would kind of just relieve the overall stress? Because what always struck me is if I know that there's a vulnerability in there and it's deployed in production environments, I'm kind of having that nagging feeling in the back of my head forever, right? Yeah.

That knowing that vulnerability is there is bad. Not knowing that vulnerability is worse because you're pretty sure that there's something in there but not knowing. And honestly, if there is something that you can do that is gonna give you some of that peace of mind, one of the most effective ways that I've seen it done is to take something like a managed application security provider.

Throw it against your source code repos where and your, um, CICD builds where you're building a container. Every time that code gets checked in, you're running static analysis. Every time that code gets checked in, you're running a dynamic application security test every time that code gets checked in and you're throwing that stuff right back into your ticket tracking system, whether it's Jira, whether it's Azure DevOps, whether it's GitHub doesn't matter.

But now where those developers work, where those developers live, they have actionable things that said, Hey, you check this SQL injection vulnerability and you check this cross-site scripting vulnerability and yesterday you should clean that up before you go back and work on whatever you, you're gonna work today. Now I know I've got that feedback instantly. Oops, I didn't even notice I did.

That lets me sleep a whole lot better at night. All right folks, you heard it here. Hey, shipping code with known vulnerabilities is roughly the cyber security equivalent of plane Russian roulette.

And it's only a matter of time before you're gonna lose. Hey Aaron, thanks for being on the show. Appreciate it.

Thanks Mike. Alright, and back to you guys in the studio.