State of Software Security – Chris Eng, Veracode

This is Textron tv. Hey, everyone. Welcome back here to techron tv.

You know, it's always a good day when I can have my friend Chris Anon with us. Chris, if, if you don't know him, is the chief research, is the chief research officer at Veracode. He's been at Veracode through every iteration, twist, turn, bend A long time.

Yeah, it's a long time actually. I know Chris even before there was a Veracode, but, uh, he always, you know, Chris is one of the most knowledgeable guys in the industry that I know, and I, I've been in this industry a long time, so it's always great to have 'em on here sharing a little bit of that knowledge and a little bit of insight. Um, hey, Chris, welcome.

It's great to have you back on. Thanks, Alan. Good to see you as always.

Absolutely. So, I, I didn't mean to embarrass you, but I do know you a long time and, um, for people out here who maybe, who aren't familiar with you and, and kind of your career path, why don't you, if you don't mind, give them a little bit of the Chris Ang story, but kind of weave it into the Veracode story as well. Yeah, sure.

Well, I've been with Veracode now for 17 years and, uh, you know, we started in 2006, uh, primarily as a static analysis company, um, spinning out of Symantec with, uh, this sort of unique binary analysis technology that was nowhere else. And nobody was ready for it, right? Nobody was ready for binary, nobody was ready for, uh, cloud.

We didn't even have the word cloud at the time, right. It was called software as a service, if you remember. Mm-Hmm.

Yep. Um, but then, you know, over the years, as people have started to realize that they need to scan it at every, uh, every phase in the SDLC and kind of do software security in a continuous fashion, um, you know, it's, it's worked out well for us as we, as we've added dynamic analysis and software composition and all the other things that you need at every phase of the lifecycle, right? And so in the process of doing that, we can then, uh, learn a lot of interesting things about the industry, which is, you know, what we're gonna be talking about today.

Um, we have so many customers and applications being scanned through our technology that we can then periodically do a little data mining, right? We can pull all the numbers that we have, every app that's been scanned and just kind of crunch the numbers, ask some interesting questions of it, but the help of, uh, our data scientists, friends at sia, um, ask some really, really complex questions about, uh, not only what is the current state of software security, but what are people doing that lead to better outcomes or worse outcomes? How are different, uh, industries performing against one another, geographies, that sort of thing.

So, um, really, really cool stuff. And we're really the only ones that can do this, right? Because of the fact that it's done in the cloud.

We have such a huge, um, customer base. We got, you know, this report we're talking about today, uh, three quarters of a million applications, uh, in the data. Wow.

So, right. But like, you, you, you just, you just don't find something like that unless you are taking advantage of, um, you know, a customer base and then just the positioning of where we're, where we're sitting. Um, so yeah, really excited to, but You know, Chris know, knowing you and, and what you do and, and, and who you are, I mean, it's gotta be like a kid in a candy store having access to that mother load right.

Of, of data and now go mine it to, to, to help people to find patterns, to find right answers to questions. It really, it really is because, you know, as practitioners, we can really, uh, it's easy to fall into the trap of saying like, well, this is what I think you should do, because, you know, it seems Right, right. Or, um, you know, logically it, it makes sense that you would do X or Y, but to be able to say, we can observe what happens when people, when organizations collectively do X or Y and that leads to outcome Z, then it's become more than opinion, right?

It's a, an actual correlation across the dataset where's statistically significant, and we can really help organizations push their program forward and do the things that are gonna move the needle for them. It's gonna make, it's gonna make them look good in front of their, you know, exec team and their board as they're talking about what they're, what they're doing with their program. So yeah, it's really, it's really fantastic and it's, it's, it's, and it's really, especially we, we've been, we've been doing this now for, I think this is the 13th version.

Um, we do this every year, right? With, with fresh data. And the first year we're doing it, I think it was a spreadsheet, uh, yeah.

In Excel with like 1500 applications. It was, you know, it kind of got bigger and bigger after that. Um, you know, to the point now where, you know, we're not doing this in Excel, right?

We, we, we, I would hope That, yeah, we use the, you know, preeminent security data science firm in the industry, and then, you know, we can ask questions that we would have no idea necessarily how to build out the, the analysis for, but they know how to do it. So we can say, okay, isolate this and tell us what happens with remediation rate. And they can do it.

It's fantastic. You know, you know what's interesting, Chris? Look, I'm, I'm a geek, right?

I've been in, in the technology for most of my life. Um, it's, it's always interesting to me how our, um, our appetite for solutions closely mirrors our ability, our abilities. It's kinda like almost a Moore's law thing, right?

You know, when we only had 30 meg hard drives, our OSS fit on a 30 meg hard drive, and our databases, you know, were contained enough data that we were comfortably able to, to manipulate with the technology, without the technology, you know, this big data capability that we have developed over the years, something like this, it would've just been a pipe dream, frankly, right? I mean, you couldn't, you know, you would've had to take the resources of several co countries to try to parse this data, right? In, in, let's say, even 15, 20 years ago.

Yeah, right? You couldn't, you couldn't, you couldn't wrap your head around it. So it, it's interesting, and, and, you know, what is that for the future as we get better at this and we apply more ML kind of, uh, you know, machine learning kind of stuff, and then of course, AI and, and gen ai, you know, what, what, what more are we going to be able to, to gather from, you know, these, these loads of data, these, you know, noodles and noodles of data.

It's crazy stuff. It's new anyway. Yeah, it really is.

But that's why I get up every day and I love to see what we, you know, what's going on in the world of tech. Um, so Chris, you know, we, for anyone who's interested, I think it was around last January when this most recent state of software security report came out, and yeah, we did a, we did an interview. It's on Text drunk tv for anybody who wants to go check it out, just go to text drunk tv, um, search Chris Ang, it's ENG, and, um, you, it'll pop up there.

But you guys recently sort of focused in on, on data regarding the financial services sector with some very interesting results. That's Right. Yeah.

And the, the, the main report kind of covers everything, right? The full dataset, some broad trends, and then throughout the year we try to, we try to release kind of little slices because the full dataset is great. And, um, you know, you get a 60 page report and there's, it's chock full of data.

But if I am the CISO of, you know, a financial services firm, that's great, and I will read that. But like, what I really wanna know is like, you know, what, what does financial services look like? I always wanna know how am I comparing to my peers, uh, maybe in other industries, Hey, how am I doing better than, you know, the healthcare guy down the street?

You know, just, you know, people just want to compare. They wanna know how well they're doing. Um, absolute numbers are, are interesting, relative numbers are better.

So in that vein, um, we try to release a little bit of a, a focus on specific industries or, or different slices that we can do. We make it a lot smaller. So this one's even, I think it comes in under 10 pages.

And we look at what are, you know, what are these industries doing well? Where are areas where they can focus on and maybe, uh, improve on for the following year? You know, and, and, and how are the trends that we saw in the full report translating down to that particular industry?

And so, you know, we find that financial services is doing pretty well relative to the others. So I'll tell you what I mean by that. So if you look at all the scans that were done over the past year, uh, you have about 72% of financial services, uh, applications having at least one flaw.

So, probably didn't surprise you, probably didn't surprise anybody. It, it doesn't seem like a high number because we know it's hard to write perfect code, right? Uh, we're always gonna have some flaws.

And, uh, they actually come, uh, they come at the top of all the industries we looked at, um, that 71% is the, is the best. And actually it's an improvement over how they did the previous year. Um, just slightly.

I mean, let, let's, let's look at the flip side of that. 28 or 29% didn't have any flaws. That's right.

That we know Of or that we could find anyway, right? Yeah. New, new flaws over the, over the past year.

Yeah. Mm-Hmm. And a lot of that is mm-Hmm.

You know, that may seem incredulous to, to some people just because of what we just said about how, how difficult it's to write code, but keep in mind that a lot of applications are small. Um, that might include like a microservice or something like that, that just doesn a very specific task. Um, and so you have, you know, you have everything ranging from your monolithic, huge platform down to, you know, a couple megs of code that just does, uh, one particular function.

And, you know, different, different, uh, different applications are architected in different ways. So, um, the number holds up, uh, uh, in my view, and like I said, it's a little bit better than the previous year, and they come in at the top, regardless of whether you're looking at, you know, any flaws or you're looking at, um, percentage of applications with flaws in the OAS top 10 or the CW 25. They, they, uh, they come in number one.

Um, so that's great. That's the good news part, right? Mm-Hmm.

6% of the apps have at least one higher critical severity within the past year. And that's fourth place, um, out of six. So they're coming in the bottom half there, Meaning they have the, they have more, a higher percentage of critical vulnerabilities than the, the top three finishers outta the six sectors.

The manufacturing, retail, and public sector even came in better than financial services when you look purely at the percentage of apps that had high and critical severity of vulnerabilities. Yep. So let me, it's A little bit of a head scratcher, right?

Yeah. But let me, let me see if I could shed some, you know, experience on this. Yeah.

And keep in mind, I've been doing vulnerability management since 2003, right? We're still secure. The company I helped found, we came out with a product called Vamp.

What's a critical vulnerability to you may not be a critical vulnerability to me for a couple of reasons. It may not be exploitable, it may not be reachable, it may be something that is not, you know, it's very isolated in our network. It may even be something I'm aware of.

I just haven't fixed it yet because I, it doesn't rise to that level of criticality to me. And, and so, you know, I remember Chris 2005, I'm meeting with, it was still Citibank at that point. I don't think they called it or Citigroup, it wasn't Citi yet, but I was meeting with one of their three global CIOs, and I said, how, how fast do you apply Patch Tuesdays?

He said, about 120 days. So they were four patch Tuesdays behind. Yeah.

You know, constantly. And I was like, but you've got these, you've got three months worth of known critical vulnerabilities out there. And he said, well, they're not as critical to us as what they can do in terms of if we applied a patch that broke something.

And so they made the risk management decision that it, it wasn't, it wasn't that critical to them. Yeah. And I'm wondering if that maybe isn't still at play here.

I, you know, I'm not saying C still does that. I'm sure we've all gotten Yeah. Better at applying patches and remediations, but I wonder if that's not, you know, factored in here.

Yeah. Um, it's, it's funny that you're talking about Patch Tuesday back in that, in that timeframe, where Were those days? I mean, a lot of your, a lot of your listeners may not realize.

I mean, now patches come out, they just work. There's not, right. It was a little bit less reliable back then, right?

To say the least. To say the least. Um, yeah.

A a lot less breakage these these days, like 20 years later, right? Or I don't remember what Patch Tuesday started. But anyway, yeah, I think, um, there, there's something to that.

And I, the, but I do, I know something about our data set that, that you don't know that which, which sort of negates that. So that conversation happens all the time, right? Like, do I need to worry about this?

Is there something else in my environment or in my risk appetite or something that, that convinces me that I don't need to take this one as seriously? And for us, there's a workflow in our platform called mitigation, okay? And it'll allow the company to go in and kind of in a structural way, make the case for why they're not actually patching or fixing this, this vulnerability, whether it's first party or third party, whatever, and then their security team, right?

So like, the developer will propose something and then the security team will sign off on it, or, you know, it's, there's like a, a kinda like that multi key type of thing, right? One person can't just decide unilaterally. Um, and so if a company has done that for a particular flaw, um, that's not considered to be an open, uh, unfixed flaw.

You know it for the purposes of our dataset. So if you've mitigated it, we consider that as if, as if you've closed it. Um, mm-Hmm.

For the purpose of this. So when we, when we talk about the, that percentage of open, high and critical security flaws, those are flaws that, um, they have not gone through that exercise for and are just remaining unpacked. So, um, So they truly are, It's interesting to see.

Now, let me, I'll tell you what, what our theory is on this one go. Um, remember that we are considering all types of flaws here. First party, third party.

And when you look at a language breakdown by industry, you find that, and you could probably corroborate this with your experience in the industry, um, financial services is very heavy Java. Yeah. Uh, 51% Java in financial service.

net is the next closest at 24% in financial services and JavaScript. You got other stuff there, but over half is the Java. Okay?

Now, if you go back to a previous report that we looked at, um, where we, we were focusing on open source, you find that 95% of the code and a Java application is comprised of third party. So it's a, a higher number, percentage of third party code than, than any other language as well. Mm-Hmm.

You take those two things together and you can trouble, you take all the, the CVEs that we know about in open source libraries and how severe some of those are, and it's no surprise that financial services comes in a few percentage points higher because they're using the languages that I think naturally gravitate to this type of, of, of assembling software. So that's like, that's the working theory. It's hard to get the why's out of these.

Sometimes we have a lot of the, what we can slice it, but it can't always tell you exactly why it's happening. That's the best guess right now is just the way that the, the apps are built. The fact that we know developers just don't update libraries very often.

Well, look, this is the whole, I mean, you know, this, this is a poster child for software supply chain security and SBOs and all of that kind of stuff, right? It Really, yeah, it really, I mean, it really kind of opened your eyes To it's the software factory. Yeah.

Yeah. Um, With third party components, Mm-Hmm. You, you get a lot of great value out of you using open source, but then if you don't pay attention to it over the years, you know, 79% of the time developers never update the very first version of the open source library they use.

So five years later, they're still using the same version. They they're still pulling it. Yeah.

Yeah, yeah. And it's accumulating Time. I mean, we saw this with the, what was the Equifax library?

That was the Equifax breach, right? Struts struts too, or whatever it was. Yeah.

Um, six months after the, the, the incident, people were still pulling the old, the defective struts down Right. And using it. Right?

Right. They super, Super common. Uh, log four J came out, right?

Log four J came out, and at the time we did a data poll and said, I wonder what percentage of Veracode customers are, you know, vulnerable to this right now. And, uh, after the patch came out, so we were trying to see how fast are people applying the patch, right? Um, and we're like, wow, like only, like about half of of of apps are vulnerable to this.

What's, what's going on? And the reason was because the, the half that were vulnerable, were using a version that was so old of log four J that it didn't have the vulnerability. It Didn't even have that, It didn't have the vulnerability yet because It was predated the vulnerability.

Right. And meanwhile, it's got 300 other critical vulnerabilities, absolutely. Extent, but like, you know, this is the, these are the patterns that, and, and I think it's getting a lot better, right?

When we look at how long it's taking people to patch third party vulnerabilities, it is getting better that these, this goes to, to other reports that we've, that we've, that we've done ab So, so I, I have a theory with that too though, or not a theory, but how I'd like to see it play out is, so the good news is with most of these third party components, like, you know, not built tier components, let's call it, that, get put into people's software supply chain these days, they, they don't pull 'em, you know, direct from Chris. They pull 'em from repos. Yeah.

And there's, you know, well, there's more than a handful, but there's, there's basically, you know, a limited amount of repositories that, you know, 80 20, right? 80% of the third party components probably come out of, you know, a dozen different repos, two dozen repos. We need, I don't understand why the repos can't build better, like, don't let them download an older, vulnerable version, right.

Or put up some sort of, you know, radioactive warning, Hey, you're downloading a, a, a bad version here. Or, you know, if, if they're, if they're making a call to something that's hosted somewhere else, and that is, is a bad one, you know, block the call, do some, like, I, I'd like to see the repos take more responsibility for the software they're distributing. Right?

But see, now you're getting to that breakage issue that, that you brought up before. Right? I know.

It was like, okay, if I'm so far, if I'm too far behind and then suddenly I'm not allowed to get it anymore, I'm, I'm gonna have break. Now, if we kept up with every minor version, if everyone kept up, you know, minor versions very rarely cause breakage, right. Um, you're not, you're not deprecating a function.

You're not probably changing how something works in a minor or a patch released. Well, your three major versions behind Yeah. You're gonna get breakage and, um, it's Gonna be, so, you know what, what Then kind of raise the hygiene also, Right?

I would give them, I would give them the capability of Yeah. You know, using, but they would have to jump through some hoops to acknowledge that Yeah. You know, they're doing it.

But, hey, what do I know? I'm just sitting here interviewing you, Chris, I, I love the idea that, and, and, and you see this happening, uh, in, in other places, that you have to really, really try hard to shoot yourself in the foot. Right?

Right. And, and what you're describing with, you know, just kind of getting a big red flag when you're down, like, Hey, are you sure you wanna do this? Here's, here's what the, you see this in programming language all the time with safer default.

Right? Right. Like, you're not gonna, you're not gonna eliminate something like, you know, SQL injection entirely because you have to, to some degree give developers the, the, the latitude to, to form queries, however, however, however they want, right?

Agreed. But you can, um, you can create a function that, um, doesn't let you triple eight stuff by default, or, or, or makes it a little bit harder to do that. You make your default, um, you know, your default, um, output encoding mechanism safe.

And if you wanna, if you wanna do something different than that and possibly cause cross that scripting, you've gotta go and set a parameter and you've gotta, you know, I love it when they call it like, unsafe equals true or something like that, because then you're right. You're actively opting into to what you're doing. Yeah.

And, and when and when something hits the fan, someone's gonna say, you, you did do this. Um, right. Give 'em a warning.

I agree. Chris, we're running low on time. Let's, let's, what else have we got in this report?

Yeah. Well, yeah, and I'll give you, I'll give you a couple more things. Um, I talked before about how, how we isolate certain behaviors and try to tie those to outcomes.

And we talked a little bit about this in the full report, um, about two specific things that we saw really, really tied to reducing the introduction of new flaws. And that's really what we are focused on with this report is how do you introduce fewer flaws, because that's gonna stop us from building up the security debt, right? That we've, that we, that we have, um, two things.

You automate your scanning, you scan via the API, which seems pretty, um, obvious in this, in this day and age, right? You build the security scanning right into your CICD Yep. Alongside all your, all your other stuff.

And, uh, that reduces your, uh, the, the probability that you're introducing, uh, flaws, uh, by about 20%. The other thing is you do interactive developer training, um, not, uh, slideware, not CBT, but something where you've got a lab environment develop and learn about what these flaws look like, how to fix them, how to exploit them even, and what it looks like. And, uh, it really tends to register in the brain that way.

Um, if you have an application where the team, the developers on that application have completed 10 or more of these very short, but interactive security trainings, they're reducing their flaw of volume by 26%. Um, and so these are things that's amazing. Like in the beginning, these are things that of course we're like, yeah, you should do this.

But to be able to quantify that and say, if you do this, that correlates with these, Know that's an amazing, i I would run with that number. That it's an easy Thing to do, right? Absolutely.

And it's something that you control, unlike organizational things, culture, But it's also I think, what people want, Chris. It's what? Look, look, we do 400 webinars a year here at text.

Yeah. Overwhelmingly, our audience who are developers in cyber and cloud native people tell us they don't want the talking head and slides anymore. They want hands on demos, workshops, learning like that, where teach me, make me better, upskill me.

Right? Right. And I, and I tell it to companies, I tell to companies all the time, please don't, you know, send your sales engineer with a slide deck on here, or your product marketing person with a slide deck on here, you're much better off getting, you know, create a sandbox for people.

Yeah. We may have less people up, but let them get, let them really learn. Right.

And, and even if you do it over three or four sessions or whatever, it makes, it makes a better developer, it makes a more secure developer, it makes a better security person. It's just, it's what we should be doing. Yeah.

It's Right. You do, right. Everyone knows that you, you know, you do one training a year, you don't learn anything.

You do. No. But let's say you do a scan of your application, you find that you've got a high prevalence of this type of issue, and then you can go immediately take a training on it the last 15 minutes and is in a sandbox that you control and you can do whatever you want in the lesson.

Sticks a little bit better that way. It's, it's more of a habit forming. It keeps security in the back of the mind for the developer.

Uh, and it's just, it's more engaging. I mean, this is why, I mean, a lot of, we used to, we used to do all CBT stuff like everyone else did, right? But then we, yeah, you acquired, you know, you probably remember we acquired, um, they originally were called Hunter two and they did the Yeah.

Uh, interactive training. And then we, we weaved that into, in, into our, um, our offerings and gave it a barcode spin. But it's just a, such, such a, um, more A game changer.

Way changer. Yeah. No, and, and it's available.

You need a quick little refresher. You want to go back to it. Yeah.

It's fair for you. Right. And you can tie it.

Agree. So anyway, so that, those, those are the, those, those are the really the big, um, you know, behaviors that I think are easy for people to do that i, that I wanted to share because it's, it's data backed. There's, there's nothing better than that.

Absolutely. Hey, Chris, people maybe want to take a peek at just this 10 page financial services sector slice. How, how can they get it?

Yeah. com/s os s, which stands for State of Software Security. And, um, and we're like almost, almost continuously producing these.

So, um, you know, I mentioned earlier, we've already done the data pull for the next, uh, full report that we're doing. It takes about six months from the data pull to kind of do the, doing the investigation, doing the analysis and all that. And so probably early 2024, we'll have the next version of, and we'll, you know, we'll answer different questions than we answered the other times, and we'll, we'll look at things from a slightly different perspective and angle.

So I look forward to that, uh, in the, in the, I look forward to having you on here and doing it again. Hey, man, Chris, it's always a pleasure to see you have. If I don't see you, have a happy Thanksgiving, happy holiday season, man.

R a's not till May maybe, I don't know if you're doing AWS reinvent, we'll be there streaming live, but, um, I'll run into you somewhere or we'll be on here. But thank Sure so much. Great talking to you.

All Righty. Okay. Chris Aang, chief Research Officer at Veracode here on text Drug tv, talking about their state of software security, uh, reports, uh, focused in on the financial services sector.

We're gonna take a break here on Textron. We'll be back in.