Shipping Known Vulnerabilities – Invicti, Frank Catucci

This is Textron TV. Hey guys. Thanks for the throw.

We're here with Frank cutucci. Who is CTO and head of cybersecurity research for invicty. And we're talking about a survey that they did that showed and I know this is going to shock some of you that developers are pushing code into production environments, but known vulnerabilities the issue of course is why and what's to be done about it.

Hey, Frank, welcome the show. Thanks, Mike. Thanks for having me.

So for a lot of folks, this is kind of scratch your head time because we are seeing you know, congressmen are starting to put out legislation that says the Dow shall not push code with known vulnerabilities into production environments and yet we all know what happens. But why what do from your perspective? What would possess somebody to put a known vulnerability into a software component that's going on nowhere production environment.

Is it just that we think that we're better or that we just don't take it seriously or what's going on? They're probably a couple of different things here at play. I can't say that developers are pushing the vulnerabilities into code willingly and happily that they're that contain vulnerabilities.

But the truth of the matter is that a lot of the code that is written today is assembled meaning that the majority of the code is pieced together with open source libraries components dependencies Etc that come from various different types of sources across the internet. Now a lot of that open source software a lot of those libraries have these dependencies and or components that are embedded in them that contain these vulnerabilities. So that's one aspect of it.

I would say that the other is that primarily the developers job is to to deliver functional code on time that works meaning that they're in. Eyes to deliver and hit their Sprints the expectations on a regular basis. Now, they don't necessarily always have a let's say an inclination or an incentive to deliver that code with zero vulnerabilities.

It's extremely difficult for a number of reasons, but their main focus or their main goals to deliver that code that functions on a timeline if you will and so that's part of the reason the other pieces they they are it's it's very difficult to fix a remediate a lot of these vulnerabilities and keep your product and or your software functioning for your end users. And those are a lot of profits and things that are tied to that is not breaking current functionality. So the ability to fix all those vulnerabilities and not break any current functionality within your software does take Time and resources those time and resources are mostly directed to money output and revenue coming in for deliverables for your clients.

Yeah. We hear a lot about shifting responsibility left for security. We hear a lot about devsecops.

What are the challenges in making that actually happen because it seems like most of the developers that I know security was an elective when they were trained and they never took it so, you know what makes us think that this is going to get any better any time soon. Yeah, so there are a couple of different again a couple of different things that I would say that are you know hindrances or or hurdles that need to be addressed one is if we're going to shift security left. There's the Assumption out there that we're already doing something on the right.

So there's already that assumption that okay. We're already having security scans and things done on the right. So now we need to like lift and shift this left.

But really what it is. It's a totally different. It's a totally different Paradigm what you have for being able to enable these developers and enable devops processes to to perform Security First of all, we can't slow down the production.

So one thing that we need to do and part of the hurdle there is to make sure that the processes that are put in place are I would say very devriendly and Dev Centric meaning that they're gonna flow at a speed and a Cadence that won't Hinder. Development speed and then what we're going to also have to do is try to work that security into the culture meaning that we're going to have to put resources into awareness and training put those resources into making sure that developers are informed and trained and how to use these particular tools and processes and these kids this can start all the way at the far left. Meaning we could have plugins into the IDE.

Right? So we're talking from real time or ad hoc development being done within the IDE that actually starts the checks, right? So do we could do this from that point and then we talk about working these processes into the flow of things like cicd.

That's where we're really talking about having these flows and hooks where we commit code the code is is scan before it passes to be merged or something of that nature. That it passes a certain threshold. Now what happens there is that we need to have I would say a very good ratio of signal to noise.

Meaning that if we have a lot of false positives. Or things that slow down that production we will have those those vulnerabilities or those findings get ignored. And so this is it's really a streamlined process that takes a little bit of time and discipline with more of a developer-centered view to to come into fruition and there needs to be that kind of buying from the top down that allows that ability of the of that security to be worked into the development processes anything from the secure design that the the tech references the documentation, you know all the way to before any line of code is getting written all the way to where it's deployed in the various stages throughout the pipeline.

So there are a lot of different unique challenges there and again, assuming that you're already doing Security on the right we shifting it left but I think a lot of organizations are still struggling to get it in on the right. So that shift left comes at A little bit of a learning curve if you will, especially with some of the tools and the different processes that could go with that. Into your point about the right side of the equation and almost seems to me at least that a lot of the security people focus their efforts on the things that they control so they invest in security operations, and they think that the development team is taking care of application security.

Meanwhile the development team seems to think the security guys are taking care of that. So this whole thing kind of feels like watching, you know, the ball constantly drop between a left field there in a center field there and nobody's saying I got it. Is that kind of important the problem I would say that that does happen especially not in probably.

The most mature organizations which is the vast majority, right? So if you look at it security maturity having a dedicated appsec function is more of a luxury of a probably, you know, more technologically forward looking company and or tech company or born in the cloud type of company. There are a lot of manufacturing and Retail and other type of organizations out there that don't necessarily have that for thought.

So having that app SEC leadership or that app SEC function specifically defined within your org does leave the opportunity for that ball to get dropped if you will where security is like the developers are responsible and the developers are like, oh security you'll take care of it later. Um, and so as that maturity increases we start to see more of a focus to understand where exactly the certain responsibility should fall and I think a lot of that responsibility Three it is is more of this like joint effort or collaboration between the groups and as we know that that's the most difficult for a time and resource constrained group to be able to to contribute to so that I think there's definitely some of that going on. How automated do you think this will get long-term because as we pointed out developers themselves don't have a lot of security expertise.

So can we embed more of the guardrails in the development process? And what does that look like and you know can the platform kind of save us from ourselves? Yeah, I think we're I think we're going to see more of that Reliance and Adoption of the automation pieces.

If we are gonna continue to be able to work at the same matter of efficiency and deliverables meaning that I think we're gonna look at this from a couple of different Scopes. So from things like SCA, right we have this software composition analysis that takes a look at a lot of the open source libraries that components the dependencies Etc. So a lot of what happens there is the automation with with some of these tools and SCA are already kind of built in and hooked into your your devops pipelines.

So yes, we're going to see more of the automation on that piece as far as okay. If there is a piece of code getting merged or or specific release that is going to be merged that contains a critical or high vulnerability. It's going to fail the build so the build fails and then that responsibility shifts over to the developer to fix that code and that could be in the form of a jira ticket or service now ticket or however that's formulated for that organization.

And that automation is going to be another words that so the developer fixes it and then they resubmit right? So they resubmit that that build and then if the bill passes or those scans pass it moves on to the next phase and I think we're That far more with things that either directly hook into the git environments meaning the git have a lab or GitHub workflows as well as software composition analysis. And then we see this, you know, static code analysis piece hooking that directly in there's some time constraints with some of that.

So we're looking at things that are quicker and more agile from a static code analysis. So we're doing a lot more things, you know with graph and we're doing a lot more things where we are looking at perhaps Iterance is of that specific test rather than doing a complete test on the code base and we're looking to cut down the time there. So there's there's some lighter scanning that's taking place there some lighter linting some code quality checks and things that we're doing that are automated and we're still going to draw the threshold, right?

So for this specific component or library or again this piece of code that we're pushing out. If it has to do with a sensitive piece of software that it that perhaps has an additional risk factor to it or that contains or processes personal information financial information. That's where we'll ratchet up that security and then fail if it has a specific finding and then submit for retest same on the desk side.

So on the desk side, we're looking at a couple of different pieces as well. Right? So we have Das that is looking at the application after it's compiled and deployed to a non-production environment.

This gives us a couple of different advantages. We're not only looking at the code or the raw Source, but we're also looking at the configuration of the hosting and the server and the environment of which that code is being deployed. So we're testing the web server.

We're testing the application. We're testing that configs of how that application is delivered and then if those have a Effect level or threshold of vulnerability we can actually fail the the push to production rather than failing previously at the build. So it's it's one of these things where kind of we're like we're moving through the assembly line of security and we're looking to say, okay.

I see a okay SAS now, you know, this is compiled and deployed with a specific configuration. We need to make sure that Das is taking a look at this to look at the web configuration to look at the the host Etc to make sure that there's a you know, it's not just the code level but it's also that that application layer as it works with the underlying hosting layer. So it's kind of a really a left to right delivering method in which needs to be scared and and it's difficult to do and that's how we end up with a lot of these vulnerabilities finding their way through is is just one of those gaps or one of those lapses that allows a specific Start to go through and have these vulnerabilities later.

Do you think the bad guys are getting better or more efficient at finding those vulnerabilities and misconfigurations that are showing up in these production environments? Yeah, they have plenty of practice. Right?

So, you know, it's funny at any given time. You will see things like you all you can't scan us. You can't do this.

We don't want to scan prod. You know, these libraries are can't be upgraded because they'll break functionality. They'll take us a week to fix.

Okay? Well in the meantime if I'm a bad guy first of all, I don't care about your rules, right so I don't care that I'm not allowed to just gain your prod environment. Guess what?

I'm doing it. Anyway, I don't I don't care that you can't upgrade your piece of software because right now I'm going out there and I'm looking at publicly available proofs of concept or I'm developing my own proof of concept code for an exploit and more advanced circumstances and we are all so looking at the the utilities and and out of availability meaning that I can plug something in that's a new vulnerability into something like Showdown. Something of that nature where I'm saying?

Okay, well find all of these exchange servers where this is accessible. Now. I have a very finite Target group to focus on and try to do things to to attack those systems.

Now people also put mitigating circumstances, right? So mitigating factors, like web application firewalls things of that nature always good idea to have those but you can't solely rely on that because there are things that attackers are getting more sophisticated about in that would be like wow bypasses, right? So being able to bypass a laugh with maybe a double encoded payload or something of that nature and it's kind of a cat and mouse game, but at the the underlying kind of Truth here is that we still need to fix that code, right?

We can mitigate it we can buy us some time. But the code behind the scenes still needs to be fixed and that's where a lot of that difficult. The comes in and actually putting those fixes in and the amount of time and/or testing it takes to do so.

Testing and scanning. What are the best practices for that? Because it seems like a lot of developers will tell themselves.

Well, I'll write some code today and then I'll scan it at the end of the week and then find the week comes there's a massive amount of code and they don't want to scan it because it takes too long. So it feels like kind of sometimes we're Our Own Worst Enemy. Yeah, I would say so so I would say best practice there is to scan everything and scan off and right.

So most likely to the scenario is to be scanning continuously would be would be a best practice and that would be everywhere. And and the reason I say that is because we look at things like, you know, I don't want to be dead horses here, but we look at like log for Jay, right? The day before log4j came out you were getting a clean scan, right?

So that's game was clean and everybody works really fast to put these signatures and things out there or to be able to dive into your code base and say hey, we recognize that this is a component that we need to find right that where is it? Well, our scans everything else, you know, everything's labeled clean until the next day when all the sudden the signatures written and or within the couple of hours. We're in the signatures written and now all of a sudden we have these kind of red flags and alerts everywhere, right?

So if you wait a week, it might be far too late, right? So what was Secure today might not be secure in three hours or six hours or not, but that and speed so the more often you scan. Especially if you're let's say doing iterations of specific code bases or scanning only what's new versus a total scan and say we do a total scan, you know at the end of the day or the end of the week, but we're gonna constantly scan what's being built and when it's being built at real time cuts down on the time it takes so one you're getting better information more accurate information as far as what might be vulnerable to you're also cutting down the time that you're waiting or the time that you would be, you know functioning as a gatekeeper per se and letting this code be produced.

So I would say that the more continuous and regular skiing that you can do on a regular basis almost non-stop. It would be the recommended path. There's also a piece to that about discovery.

Um, we look at a best practice for Discovery meaning that we want to know everything that is is out there. We want to know everything that's responding to a get requests on our Network or our ephemeral Edge these days right? So we want to know what is out there.

What is being spun off these services are constantly being spun up and spun down these these these these containers these environments kubernetes Etc at work constantly ramping up apps and attack surface something the load goes up. The these hosts are fired up. This new code goes out something of that nature.

We're releasing multiple times a day. We're no longer have this static code base where we're releasing, you know, once a month or once a quarter, right? So the codes changing the the environments are changing the attack surface is changing and really the only way to know that is to be looking almost in real time.

Closest you can get to keep those scans and keep that Discovery going in a real-time basis. And folks I heard it here scanning is a lot like other preventative health measures that you might take in your life and you're much better off having them taking them early and often as possible. Hey Frank, thanks for being on the show.

Thanks, Mike, and I would be happy on any follow-up if anyone has any questions feel free to reach out. All right guys back to you in the studio.