What’s Required to Make DevSecOps Really Work – Wayne Jackson and Brian Fox, Sonatype

This is texturing TV. Hey guys. Thanks for the throw.

We're here with Wayne Jackson is the CEO Persona type and we're also joined by Brian Fox. Who's their CTO Wayne? Welcome the show.

My pleasure. Great to be here Brian. Yeah, thanks for having us our pleasure.

See you guys just passed the hundred million dollar annual revenue Mark. So that's a congratulations to you folks. The question, I would start out with Wayne is you know, we talk about devsecops and security software.

More now than ever. But my question is is it really happening? Because I feel like a lot of times it's kind of like that Sunday sermon everybody went in and not at their head and said, yeah, that was great.

But then, you know, like a couple hours later life gets in the way and they forget about it. So, you know, is it real and what are what are you seeing and people do well? I think it is becoming more real I mean is as you may know I was the CEO of sourcefire back in the 2000s and And so lived on the network side of the of the cyber world before coming here.

I think there are still way too many abstract functions that live in a crusty old Silo where they're scanning things after the fact and passing reports over the boat, you know over some fence and hoping that behavior change but we are starting to see some real and impressive modernization of the outside function where absec starts to weave itself and it's ideas and it's principles and even some of its tools into developer oriented workflows and into the process of innovation and in those organizations, we see real and genuine Improvement both in outcomes and in cultures Brian we talk a lot about shifting left these days and there seems to be a little bit of a debate about whether we want to shift left all the way to developers or do we just want to kind of lean left into the devops workflow and embed more Automation in there to handle security issues. What is the right approach that we need to push this all the way to the developers who seem their resist sometimes or do we need to embed it into the devops workflow or maybe a little bit of all the above? Well, certainly it has to be embedded into the devops workflow.

But you do need to push it to the developments the developers and and maybe even a little further left of that. If you're thinking about the supply chain, you know, ultimately it's the developers who are going to have to act on any of these findings regardless of if they're SCA component level findings. Custom code level findings, right and so not involving them not empowering them just perpetuates that friction that you alluded to and so yeah, you really do need to get all the way over, you know, and the companies that we see that are most successful are the ones that integrate.

Their application security practices with development. They may not always be You know literally as integrated as like I think the the devops the original mentality would have that the same team is full stack in that but the more you can get application security into it and further and educate the developers and Empower them with the right tools the better off you'll be because you don't have to chase that out later when the inevitable problem occurs, and you're not prepared for it. Wayne we've seen a lot of high-profile breaches in the last year that by an Administration is issued an executive order for federal agencies is all this coming to a head.

Do you think Enterprises are following suit and they're all going to be reviewing their software Supply chains more aggressively as a result. I mean, I think maybe this is the year of evaluation and maybe next year is the year of doing because ultimately people need budget to drive this right? Um, you know, I I think the tide really started to turn with Equifax back in 2017.

I think that was the sort of the Watershed event where Not only was there a high profile breach but senior level Executives lost their jobs and and so, you know, we we saw a meaningful shift in our business and and and lots of ways that are measurable in accounting terms like, you know, you know time and procurement stage that sort of thing. Um, yes, the white the White House's guidance has brought a lot of attention to it. Obviously log4j was was a watershed event, probably more significant than Equifax.

But I think this is you know, SCA in particulars is crossing that sort of Chasm will being a must have part of the portfolio. You know, the thing that that I think is interesting and Brian can touch on this is that we're also seeing a shift in how at least, you know, the black hats are thinking about software Supply chains and software supply chain vulnerability. It's it's not just about whether a zero day disclosure and the existence of log for Jay is something that represents an attack factor.

It's it's the notion of actually creating a tag vectors by malicious commits and and other techniques like type of squatting so I think This is really kind of early days in the way that we think about software supply chain integrity. how smart are the bad guys getting Brian because if I look at the number of exploits most of them seem to be known vulnerabilities that we've known about for the better part of a decade and I wonder if the bad guys are saying why should we work so hard when we can just exploit what we already have and maybe what's the point going all the effort to track down a zero day vulnerability when it's already pretty easy for them to do what they need to do already. That that blind spot is one that I think a lot of the industry shares.

It's one that I've been trying to shine a light on frankly since 2017, you know in 2017. We saw the first of what a really the open source supply chain attacks and and interestingly some of those early attacks were focused on open source Publishers. The the malware that was introduced was not trying to steal credit cards Social Security wasn't trying to crypto mine.

It was literally trying to steal the password to publish things to the npm and python repository. That was a dramatic change intact and escalation of the significance of this and and we've been talking about it and tracking it year over year. It's been part of our state of the software supply chain reports, you know, it was it was up 600 50% last year and we're up this year another 500% on top of that and these These are intentionally malicious components being injected into the repositories designed to do harm as soon as the developers download them.

And you know, I find so many people in the industry are completely unaware of this they're worried about You know trying to inspect the software before they ship it to make them safer, which is is great. But in a world where your developers accidentally download the wrong component, they get a back door that's completely invisible to most of the application security portfolio. And and you know, I I like to use analogies.

I think there's one that's relevant for this if you if you think back to the 90s when browsers were inherently vulnerable just going to malicious website was enough to get you attacked and the reason why they were successful for a long time was they relied on the behavior of people saying, you know, they go to the website they go whoops. This isn't the website. I meant to be at they hit the back button go to the right one and never give two thoughts about the fact that they might have just been back to work.

Well that's precisely what's happening right now developers can potentially download the wrong thing because the malicious component has an underscore where the legitimate one has a dash. It's a very simple thing to confuse. And they get backdoored these components today are not even trying to pretend to be the legitimate component.

They're not going to compile. They're not going to run the test. They fail almost immediately and they're relying on that naive behavior of the developer saying whoops.

That's not the one I meant to have. Let me go get the official one. Then they carry on then they check that code in so an application security program designed to look even as early as the continuous integration phase probably never sees that attack because it never got checked in right in the meantime, the developer machine might have been compromised Keys stolen all these types of things.

And in fact earlier this week, we found more that we disclosed in Python that we're again trying to steal developer credentials. Right? So the the attacks are actually quite sophisticated.

Some of them have been multi-month efforts of social engineering to get into the right place. I just think the public is still woefully unprepared for You know the the old boring things like log 4J which was in fact just a bug latent for a long time. And that's the sad state that if you're not able to deal with that you have no chance of dealing with the malware situation that's happening right now.

Wayne you guys are working with a lot of the open source community and a lot of these issues and there's been efforts to launch or at least raise money to help the open source community and the maintainers Securities environments But ultimately what's the real challenge because it seems like you know, a lot of the maintainers are saying it's not necessarily my job to be in charge of security and passion all this stuff and when there's probably only two or three of them actually working on the project. So how do we kind of make open source software safe? I'm not sure you can make any software safe.

I think you know, we, you know vendors like us can certainly provide tools and assistance and support for free to those interested in leveraging them which which we do and have for years. I think raising awareness about the importance of hygiene as is a is a project quality objective is something is another one of those cultural things that open source committers and maintainers need to embrace and I think we need to get better at helping folks pick better projects, you know, they're you know for any given piece of functionality there are dozens and often hundreds of similar looking open source projects and helping developers and organizations, make more thoughtful choices about which projects to use I think is another one of those really important objectives that that we all should have, you know, and you know, we we talk and talking here a lot about security, but you know, ultimately what what organizations are Going to get from open sources better Innovation faster Innovation leveraging other people's IP. And and you know picking the right projects having fewer similar things is really critical to long-term Innovation because it represents a smaller surface area to maintain it improves things like develop portability.

And so and so that's another one of our really important research goals. Yeah the dovetail on to that if I if I can, you know in the wake of love for Jay, you know, I think a lot of people woke up and and had a maybe a slightly Panic need your reaction, which was something to the effect of surprise that so much critical infrastructure was in fact built on top of Open Source, I misunderstanding that everybody in open source, we're volunteers and therefore inexperienced which is just clearly not true. And the expected reaction is well.

We need to fund them. We need to give them tools. We need to educate them.

Yes to all of those things, but that is going to take a long time before. That really, you know perpetuates across the ecosystem and part of the problem is all of that Focus. I think has a lot of companies sit sort of sitting back waiting for the industry to solve it when in fact, they can start building the organizational understanding of the components.

They use today. Our customers have been doing it for over 10 years and we've shown in the case of log4j that when you do that when you already know what components you're using when long for J happened they were able to remediate tens of thousands of applications in the first three days. It was effectively a non event.

If you already had, you know a recall like capability and and I worry that while yes, all of these efforts to help open source get better our important. I I just worry it's causing people to be passive about this and you know, a hyperbolic example is if you think about auto manufacturers and and the airbag recalls, imagine if instead of them saying you know what, Is all the cars that are affected we're fixing them. We're making this happen instead.

They said you know what we're going to give more money to the airbag manufacturers that we're going to teach them how to do a better job next time the next versions of the cars will be better. And that's it. Like they would be sued into Oblivion like we would laugh in their faces if that was the approach that they took because we expect that physical goods and and food manufacturers can do recalls.

Why do we allow and accept that to be the state in our software? You know, so it is kind of a ridiculous situation that we find ourselves in and you know, we're trying to shine a light on that because I think if people think about it that way it's pretty obvious what you need to do as a company consuming open source. Brian speaking of obvious things.

We give people tools sometimes but even when we give them the tools they don't always use them and you'll hear people complain that it takes too long for the analysis to run or the scan takes too long and you know, then they don't do it and then the next by the time they do get around to it. The code base is huge and it takes even longer. So how do we kind of put this in a way that the developers will either actually do it themselves or is there some way to automate it on their behalf?

But can we get to some sort of workflow that they resist less? Yeah. I I think it's all of those things.

There are workflows, you know, you need to use tools that are designed for the way modern software development works, you know tools that take 24 hours to do a scan and produce a book of findings are not modern right tools that will produce the right effect our integrated like we talked about before further left into Development infrastructure, you know in pull requests, for example, that's some of the stuff that we do. We find that when when findings whether it's code quality or security or even SCA dependency problems are introduced in the pull request when the developer is in the mode of collecting feedback, usually from their peers, but why not from automation, it's 70% more likely to be fixed right in that moment because they're in that motion of I'm working on this. I'm looking at how to make it better.

It's no different. If a tool tells them to do that versus their trusted peer right they're going to fix it before it gets merged. And so, you know, you need to be looking at tools from all different dimensions that are really focused on that aspect of it because that's the way to get things fixed.

You don't show up later with a book of things and tell developers all the things they got wrong. then you know when it also seems to me that there's something wrong with the way we go about budgeting for application security the developers somehow think that the security team is gonna fund that and do something with that and I think the security team thinks the application developers are going to do that because from their perspective they're like, hey, we only have control over the things that we own such as you know Network firewalls or whatever and since we know Own or implement the application security tools then you know that belongs to somebody else's budget said we need to have an adult conversation about how to fund application security. that's great point and and I think that there is not a buying Center and a budget associated with especially these modern application security efforts is a is a signal of an Emerging Market.

You know, what you touched on earlier this this isn't a mature Market where there is a buyer and there is a budget that's well understood across pretty much every Enterprise that said the majority of our business is funded by the security budget. There's often a negotiation between the security function and the business units that are that are leveraging or that need the leverage of the Innovation. We also see in some more Progressive organizations the establishment of a of a devops tooling and infrastructure function, especially in organizations that are going through digital transformations.

So I think we are trending toward more. Well defined budget and a more well-defined buying Center, but my guess is that that's still a fairly long way off. Brian we hear a lot about AI these days and Security will AI save us from ourselves someday, or is that just all wishful thinking?

Um, it certainly a powerful way to get to automation. I mean we've we've used MLA I Technics to try to identify. Well not try but to successfully Identify some of that early malware that we talked about.

You know, we've we've blocked and reported as of yesterday somewhere north of 88,000 packages in the last year. He's intentionally malicious things and we did that by building models of what is typical for a project similar to credit card fraud protection to understand. You know, when a transaction is happening in real time that it seems uncharacteristic for this this project or this this person or this type of person we did do the same thing and it's it's been quite successful and we're looking at other ways to leverage that technology.

You know, you see Um code authoring tools trying to do similar things that are using, you know, AI to learn the most common patterns of code and auto suggest them. So yeah, I think that is going to be you know, a significant way down the down the path of getting of making this better much like, you know, some of the early Safety Systems in cars, you know that are sort of riding shotgun trying to help, you know, keep you in your lane and break if you don't, you know, I think I think that's the equivalent of what we're gonna see for those for the first sets of tools beyond that who knows me maybe we get to a place where the machines take over and they're writing their own code, but we still have a lot long ways to go before before we're talking about that. Yeah, I I would just add to that, you know the the scope and scale of the open source ecosystem.

Is sufficiently vast that I don't think you can accumulate enough knowledge without using Ai and ml to even think about solving some of these challenges. I mean, they're literally millions of Open Source projects and millions of Open Source committers and Brian currently if I'm wrong, but we're processing something on the order of 21,000 release events a day. And you know in order to help an Enterprise think about the Integrity of their software Supply chains.

You have to know the interesting attributes of Literally every piece of Open Source and every version of Open Source. That's observable. I mean, so these are just you know challenges that defy manual or human curation Brian last question.

What's that one piece of advice that you would give to people about how best to get started or the converse of that is what's the one thing that makes you shake your head day in and Daya there's a lot of them, but I think I think I you know, I always like to be provocative I would I would, you know leave people with the following tabletop if I told you about a new vulnerability disclosure today, let's say the next log for Jay it was gonna happen tomorrow. Do you already have the ability to know exactly where that's used in your applications? Can you tell you know in your portfolio anywhere so many people couldn't even do that can you know exactly which applications and can you track the remediation and then how would you know or block your developers who made a typo mistake from getting a back door dropped on them?

Download those components if you don't have compelling answers to those right now, you need to get started and you don't need the weight for the open source ecosystem to to get better because as long as the humans are writing the code mistakes are going to happen. These will always be problems. No matter how much we educate them and how much we fund them as the consumer as especially businesses that are consumers of this you can do something today.

So if you can't answer those questions and you need to get started All right. Hey gentlemen, thanks for being on the show and sharing your knowledge and expertise. My pleasure our pleasure.

Thank you so much. We appreciate it. All right and back to you guys in the studio.