EP 258: Is the Software We Create More Secure? Veracode’s 10th Report

com. And you're listening to another DevOps chat podcast. Today, I'm joined by Chris Eng, who is chief research officer at Véracode.

And we were talking about their latest report, The Veracode State of software security. This is volume 10. Chris, welcome to DevOps Chat.

Thanks much. Great to have you on. Just wanted to start off by having you tell us a little bit about yourself, what you do at Veracode.

And for those that don't know, Veracode, just a little bit about Veracode. So as you mentioned, I'm chief research officer here at Veracode. I've been here for 13 years now and my teams are responsible essentially for building the knowledge that goes into our products.

So we have a number of different products that help our our customers identify vulnerabilities in their software. And my team essentially identify what are the patterns that we're looking for. What are the common mistakes that developers are going to make?

What are the important things that we should scan for in each language and specify to our engineering teams how we do that? I also own product security for Véracode, so making sure that we deliver secure products to our customers, our customers, or basically anybody that builds, buys or deploy software. And so we help them build programs around securing that software at scale, using a number of different technologies, including static analysis, dynamic analysis of recomposition and so on.

So we work with anybody that that that deals with software, which is pretty much everybody, just about everybody these days with digital transformation, no doubt. Thank you very much for that. That background on yourself as well as about their code.

Well, let's start out with. You've been doing this report for 10 years and your longevity with the company being able to oversee and be part of that over that decade process. What kind of things have you learned since you started out, you know, kind of the size of how many how many applications and things that you're scanning to what's happening today and any trends that you've noticed from that?

Yeah, it's been a great process to do this every year, primarily because we're in this unique position where we actually can see what's going on there in the industry. And when we started doing this in volume one, we only had about 15 hundred applications in the data set. And then fast forward to this year.

We had eighty five thousand unique applications. So 50 times more than we had before. And I'm not aware of any other study that has quantitative information on software security that that's anywhere close to as big as this one.

So in addition to the size of the data set getting bigger, we looked at fixed times and we found that those have roughly stayed the same, which is sort of a little bit depressing. But that being said, there's a lot more software now than there was before. So there's a lot more to scan, there's a lot more to secure.

And so you only have so much bandwidth to to do that stuff. So we saw fixed time stay roughly the same. We did see that over that 10 year period.

There are fewer apps that have no flaws. So that's more of a factor of our capabilities than anything else. We can detect more than we could before.

But we did also see that there are fewer flaws sorry, fewer apps these days with with no high severity flaw. So customers are getting better at identifying and remediating high severity flaws. We looked at a number of different angles, including compliance trends, things like that.

But for the most part, I think that the fixed times is kind of the interesting part. And we dug a lot more into that and explored some of the factors that that lead into fixed times. If you just think about what's changed over a decade, I mean, 10 years ago we're thinking about sequel injection, cross site scripting.

Those are the kinds of security flaws, at least a lot of what the focus was. Things have changed drastically since. Yeah, we are still seeing the vast majority of the same four categories that we saw 10 years ago.

And I don't think anybody who who does apps that go on a daily basis would be too surprised by that. These things are really easy to fix tactically, but when there's so much of it and it spans across such a huge application inventory, it does take a lot of effort to close these. So we still see sequel injection at roughly the same rate that we saw it 10 years ago.

Other categories like process scripting have actually gotten more prevalent and we've seen some go down. We've seen like buffer overflows and numeric errors reduced in prevalence. Part of that I think is is due to the change in and programming languages as well.

Right. There's a lot of that that stuff that's prevalent in native code like C++ and we're seeing less, fewer applications being written and those languages today. So I think that's contributing more to the prevalence differences that we're seeing.

I to have service oriented architectures, micro services, things that promote reuse. Of code maybe help some of those things. Maybe it won't.

We'll see. Yeah, I certainly could. I mean, anytime there's re-use, you know, you inherit the functionality, but you also inherit the risk.

And that's you know, that that's that's sort of the trend is that we as we've seen people start using a lot more open source than before. Again, you get the functionality, but you also get the risk. And so you introduce new vulnerabilities to your applications that way.

So certainly with code reuse, like you mentioned, you could you could get that effect as well. Mm-hmm. Potentially even infrastructure's code.

There were some really great things that came out of the report looked have you talked about some of the the applications accruing debt? How many folks are driving that down? How did that what kind of effects that had?

Yeah, we we took a look at security debt for the first time in this report. And you know, most most people have a concept of technical debt, just, you know, things that get old and crusty in your software over time and maybe architecturally or things that you meant to go back and fix, but you never did. And security flaws kind of have the same tendency to build up over time.

If you think of it like financial debt, right. If you charge something on your credit card and then you only pay the minimum amount every month, you're gonna be paying a lot for a long time. And when you add that up, it's gonna be a lot more than it would have just been to pay off your balance right when you accrued it.

Right. And so security debt, when we look at security debt from that angle, we look at our application's kind of accumulating new security debt over time or are they driving it down? And when we look at the number of floors that are kind of left unfixed in an application, we kind of view that as the security debt.

Right. And, you know, there's only a certain amount of capacity. Right.

It's very difficult for an application that let's say it's been accumulating security issues over many years to just say, like, hey, we're going to pay that all down today. Right. You'd have to dedicate a lot of engineers to doing that.

You have to like, you know, really focus all your efforts to do that. So it's not typically very practical to do that all at once. But it is important to kind of be measuring whether or not you're, you know, fixing more than you find or finding more than you fix, because that's kind of directional.

Right. And so we found that, you know, nearly half of applications are are finding more than they fix. And so essentially they're they're accruing that debt.

So they're never going to they're going to climb out from underneath that unless unless they start to reverse course that. Right. Put more effort into fixing issues so that they over time reduce that debt and get to a point where, OK, now there's nothing kind of outstanding.

And and you can then make a rule that says, you know, we're not going to allow this application to be promoted to production or whatever. If we accrue new findings. But that's that's rare.

Right. So barely anybody is there yet. And so we looked at how different DevOps tendencies or methodologies practices affect security debt.

So I think those are some of the findings that I like to talk about a little bit. Sure. Let's go to that, because I'm very curious about as we shift left.

We're trying to get security and early, ideally earlier, all of those things hopefully fixed that before it's ever goes into it. Q A function or an automated testing, right? There's all those studies that are kind of show that it costs a lot less right when you fix it earlier.

And that makes sense if you're if you're fixing a flaw as a developer, writing the code, if you can, you know, sit there and tell them what they did wrong before they even check that code into the repository. It's gonna be a lot cheaper than if you find it in a penetration test, let's say after the code has been deployed. And then you've got to go figure out the root cause and then you've got to find a developer that understands that code and get it in their backlog.

So so that's that's kind of an accepted understanding that that it costs more the later you do it. That's what we will do. And pointing fingers, truce codes, it is a day that the servers or the code, you don't know exactly who is trying to tracing it down.

And so we wanted to really say, like, does devops make a difference in and how quickly we can fix things and how much security debt we accrue. And so sitting from where we are, remember, customers submit their applications to us. And DevOps is is more it's not just automation.

It's also like culture and process and a lot of things that take that kind of feed into whether you're doing devops or not. But from our vantage point, we can't see culture, we can't see process, we can't see automation. We can't see to what extent automation has been incorporated into an applications testing cycle, for example.

And. We use scan frequency as a proxy for whether an organization is using DevOps. So by that I mean how many scans do they run per year on an application?

And so you have a lot of applications that are scanning literally once per year. Right. Thirty six percent once a year.

And if you think about how quickly software is is changing and how many features are getting added every week or every month, like once a year is, you know, that's not that's not very good. On the other end of the spectrum, you have about point 3 percent of apps that are scanning basically every day or more. So you know that that's probably being done by automation.

You don't have a person sitting there. Suddenly the application, hopefully. And so if you look at there's some charts in the report that kind of show what that what that distribution is.

But essentially, if you scan once per month compared to if you scan every day, your median time to remediation is significantly faster. So 19 days, media time to remediation for a floor if you're scanning daily versus 68 days. If you're scanning monthly or less.

And so we then kind of broke it down into different Neoh, even even more buckets. Right. So we did kind of want what is one to three scans per year look like four to six seventy.

Well, you know, and so on. And it's kind of hard to describe this just just fire audio. But if you pull on the report, you'll see all these kind of like iceberg charts.

And so you'll see like pink and blue. And there's the pink represents the dead. It's kind of below the line.

And that's kind of shows the number of findings, Parap, that are outstanding. And what you'll see is that for the apps that are scanned more frequently, less debt accumulates. Essentially, the teams are able to get after that debt faster.

And while it doesn't go away completely again, because this is this is aggregating all of the apps we have it it accumulates less. And so we see a direct correlation between that scan frequency and those fixed time and the security debt cumulated certainly makes sense automating that. That's going to show up in your numbers around the 19 days to fix scanning once a day.

Is there a way to tell if it's better to see scanning happening more frequently than on a daily basis? Is that helpful? Or are there other things that help again, help the shift lacked?

You know, I think once you get to a daily basis, you're probably I think there's going to be some diminishing returns after that. I mean, imagine you did a full scan. Every time somebody checks something in, you'd just be getting a lot of it from you wouldn't be able to act on that.

Quicker than, you know, I think that the span of a day or a few hours so, you know, scanning every few minutes really wouldn't buy you anything. Mm hmm. So another thing that we looked at that we thought was interesting would be scan cadence.

So not so much how frequently were you scanning, but how regularly are you scanning? And so you can imagine. And again, there's a great diagram.

It's my favorite diagram in the report, actually. It's just a bunch of dots on a chart. But what it does is it maps out.

Every dot represents a scan. And so if an application is scanning on a very steady basis, you would see evenly spaced dots across the course of the year, whereas if you were scanning and kind of a burst fashion. So basically a lot of activity followed by no activity.

Then you would see a clustering of dots followed by just a bunch of whitespace. And so we actually calculated that cadence for every single app in the data set. And then we grouped them into buckets again.

So of the ones that were either scanning on a steady basis or a burst basis or something in the middle, which we called irregular and irregular, it's basically just a bunch of mini bursts. So what we wanted to answer was how does that scan cadence affect security debt? Are you less likely to accumulate debt if you're scanning us steadily or in adversity fashion?

Right. Which one's gonna be more effective at wiping that out? Because you could see it going either way, right?

You could ceti you could just people could get just used to seeing results and they get to the point where they ignore that maybe. Whereas Bursey like, OK, why paying a lot of attention to this? We're going to focus all our efforts on this and then drive it down.

So there was actually a question that we didn't know what we're going to see. But when we did break it out, we found that when you do the burst, you scanning, you have all that whitespace and you just put a flurry of activity over time, like you're just that security debt that that accumulates is just massive. You see this huge increase in the amount of htink which represents the the the findings that are not addressed.

Whereas with the steady and even to some extent the irregular scanning, you see the security debt grow a little bit, but then you start to decrease. And so the curve is actually going in the right direction towards towards the end of the timeframe that we're able to chart out. And so that was another good finding for us.

So what we can do is we can actually take those conclusions and advise our customers. And really anybody that's building software that if you want to reduce security debt, the data suggest that you should scan frequently and that you should also scan in a steady basis. You shouldn't you shouldn't ever let up.

Right. It has to become. And that makes sense.

Right. Anything that we make a habit of tends to just become part of the way that we do things. And so it was nice to have some data that actually reflects what we what we think would be true actually does turn out to be true.

It's kind of like a brush your teeth daily, right? Not the day before. Yeah, exactly.

That's good. You know, it also occurs to me as this burst, the style is there's really no predictability around how long security floss can exist in your code because you may find it, fix it now, but it may be if it's another three months or whatever before you scan again, who knows women, it gets fixed. Could live about there for a long time.

Right. We we we actually did find it's one of the you mentioned that the the highest probability for a flaw to get fixed was in the first month or so. And so that the longer you wait, the less likely a given fly is to get fixed.

And so essentially, developers are prioritizing kind of like last in, first out, they're more likely to prioritize something to get fixed. If it's kind of fresh and that's not what we want to see, right. We want to see developers fix things that are more important to see them fixed, more severe items or items that are in applications that are more critical to the business or items that are more exploitable than others.

But when we measured all of those and we looked for patterns that would suggest that they were prioritizing in that way, that's that sensible to a security practitioner. We found that that recency like how how recently was it found was was really the highest correlation in terms of whether something was going to get fixed. So we have to do a better job of prioritization, not just, you know, understanding what a security person would do, but getting a developer to adopt those same priorities.

You're actually not measuring the human psyche element of this, but I have to believe you build up such a mountain. That debt at some point gets too hard to grok, understand, fathom. And there's probably even abandonment rate, it seems like two big questions still with what's on our plate right now and here's the scan results and jump on it.

It does. And that's not that's not uncommon to see customers adopt that type of strategy that the idea that. All right.

Well, I'm starting this program now on day zero. I'm responsible for abscessed now. And you know, anything that happened before I got here?

Well, that's not my problem. Let's just focus on not introducing new flaws, which is fine. Acknowledges the new flaws is great, but that doesn't help you with anything in the past.

And ultimately, it's all risks to the application, right. You can you can get attacked any in any number of these ways. But we have seen we've seen big customers try to take that approach.

And, you know, it's it's not advisable. You you have to. You have to chip away at that debt, even though you can't do it all at once.

Maybe you you work in your security sprints into your lifecycle. You find a way to pay it down over time. And eventually you get to where you want to be.

But you cannot just ignore it and say no new floors going forward. That's it's not you know, it's not going to get you where you want as far as the debt. And it costs more to fix stuff later, too.

The longer we wait. You know, I imagine you've got a library to patch and that libraries, you know, one year out of date or five years out of date, it's gonna it's gonna cost you a lot more in terms of effort to fix them. One that's five years out of date.

Things will inevitably break. So. So the cost of fixing also increases over time.

There are hundreds of findings, if not more, in this report. Not to pick out just one, but it stood out to me that there was one stat in there of C++ carries three to five times more unresolved flaws than dot net over the same period of time. So this is to the point of languages make a big difference.

And why is that so? Yeah. You know, we did break it down by language and just kind of took a look at how much security debt accrued in each one.

And it's not really to say that. Okay, well, if you're using C++, you should rewrite that and you know, and dot net or some other language that's not practical for most people. But what it does show is that certain languages are more likely to accumulate debt over time and a sport not so much.

Look at that, the raw number of flaws, because some languages, you know, just r are inherently more secure against certain classes of flaws. Right. It's a lot harder to shoot yourself in the foot in certain languages than others.

But in looking at the shape of the curve like is the security debt tend to increase or for certain languages, that that's kind of that's something to look at and do at least be aware of. If you have certain parts of your application imagery written in, you know, BHP, for example, you should be aware that, you know, those those apps are probably going to be more likely to accumulate debt than, say, the dot net ones. And so it's something to to take into account as you're as you're doing your planning.

Ok, very good. Well, we could talk about this for maybe days as so much information. So we don't leave everyone with a mountain of gosh, there's so much in here to learn and understand each other.

Two or three takeaways. If you're a developer, lead developer, development manager, architect, sitting out there listening to this, going, OK. So what do we need to know?

There were a couple of takeaways she would suggest. Yeah. You know, I'll kind of reiterate some of the some of the things that we talked about.

But but essentially that security automation, especially if we look at scan frequency, it is definitely lagging the adoption of DevOps in general. Right. Dev ops is kind of just taken off like a rocket ship and security automation.

Like I mentioned, only 5 percent of the apps are being scanned weekly or better. And so there's there's some catching up to do there. The good news is that when you actually do that and when you actually do that, frequent and steady testing, you will probably get to a point where you can start chipping away at security debt and and eventually over time drive that down so that you can take a strategy of, you know, new floors and keep yourself at a at a clean pace.

And the last one, I think there's a conversation that needs to happen between security teams and developers in terms of prioritization. You know, I talked about how developers are not prioritizing in really a security appropriate manner. The recency is appearing to kind of outweigh every other factor.

And so there's, I think, some improvement that most teams could make there where, you know, even with the same amount of bandwidth to fix flaws, they could spend their time fixing things that that are more important to be fixed as opposed to the ones that are just appearing most recently. So I think those are some of the takeaways. And like you said, there's a lot in the report.

It's a pretty interesting read. And I would encourage people to go grab a. Copy.

And read through it. And very well done. Mind if I do say so myself?

Very well put together. Chris, I'm working, folks. Get the report.

It's on our Web site. So VERICA dot com and it should be on the front page there. And get themselves a 50 page PDAF behind it.

Okay. Perfect. We'll include a link in the description for this episode, too.

Excellent. Well, thank you so much. Pressure you're being on, Chris.

Yeah, my pleasure. Great to have you again. Thank you.

My guest today, Chris Eng, who is chief research officer of very code. And of course, thinking you are listeners for joining us today. We know your time's valuable and it's a great topic.

Security's important in having this information if it's very valuable to all of us. This is Mitch Ashin with Dubost Needs Lessons to another Dubost chair. Just be careful.