Impact of the National Security Strategy on Appplication Development – Varun Badhwar, Endor Labs

This is Textron tv. Hey guys, thanks for the throw. We're here with Varun Baris, who is c e O for Endor Labs, and we're talking about the national cybersecurity strategy and the implications that we'll have for developers in DevOps teams.

Varun, welcome to the show, Michael, great to be here. Thanks for having me. Again, There's a long way between proposals and actual law, but when you, as you look at this strategy, it clearly seems to be saying that organizations will be held accountable for not just how they manage data, but the security of the applications that they create.

And that seems to have some rather, um, major implications for organizations that build software. So what's your sense of where we are on this journey? What should people be taking from this document and how long before it becomes real?

Yeah, I think it's a great question. It's a pretty loaded question, right? I think, uh, first and foremost, I think the train has left the station as it relates to a conversation around, or the demand for software transparency.

I think for far too long we have deployed software from vendors from other third parties where we don't quite understand the ingredients within them yet. Log four J was a great wake up call because when we had to chase down and figure out if we were using log four J, not only in our first party code, but if we were running log four J and anyone else's code, it was a nightmare. And I don't think anybody wants to go back and send a thousand emails to vendors asking, are you using log four J?

Are you using Log four J or not? So a is it, is it relevant and is it important? Yes.

B you know, is it now in an aspirational state, at least from a piece? Yes, right? The trains left the station, but it hasn't really figured out what time it reached the destination.

And I think the hardest part in these transformations is how does this not become yet another compliance checkbox? How does it actually move the needle in terms of improving security of software? So a couple things, Mike.

One is, you know, should the producers of software take on more responsibility? Yes, they are the suppliers. It makes a lot of sense, but here's where things get really hairy.

Mike now is an open source developer, somebody who's just a good Samaritan who put on some hobby code out on the internet that everybody else decided to use. Is that person a supplier? Likely not, right?

So I think there needs to be more clarification on, you can't hold somebody who gave you no warranty, no ss l a software, you know, use it your own risk, um, to be held liable to support it for you. You know, going back to the Log four J incident, so many companies sent the maintainers of log four J emails requiring them to respond to their seven questions or 10 questions. It made no sense.

So I think we need clarity on definition between who is a software supplier and who is not. And I think in my mind the, the most complex thing in this is people build software. They rely on open source software from other people.

Those other open source suppliers depend on software from other open source maintainers and suppliers. And it is a very convoluted, um, graph of software usage and reliance. So who does the buck stop with?

Does it stop with somebody who provides cost driven, you know, through a commercial transaction software? Absolutely. But I think that needs to be clarified and the teeth are gonna come in terms of timelines for enforcement and consequences of not look, not meeting those guidelines.

And I think G D P R was a great example of how to force consequences when it affects your top line, when it affects you is revenue. That is when people will take this seriously. You think people will stop incorporating software from quote unquote hobbyists if there's only one or two folks who are building the open source software and they'll just be more careful because of these guidelines?

And that may be a shame, but it may just be a fact of life. Yeah, I don't think that's the right answer. I think the right answer is we all have to embrace open source.

I, I think, I think that is what fosters innovation speed to market and that's how companies stay competitive, right? If a large bank stops using open source software, the next FinTech startup will and they will outmaneuver them. So I don't think that's the, that's the choice, but I think the, the truth is somewhere in the middle, it, nothing is free in this world.

Stay with open source software. We're gonna use open source software, great. Invest in resources and programs to understand what is this open source software, how good is the code quality?

How well maintained is it? What would you do if the maintainer walks away from the software tomorrow? You know, how do you keep up this software in your environment?

Uh, I'll, I'll give you a, a statistic that is just eye-opening Mike to me. Yes. We all talk about the top line number, right?

80 to 90% of code in our environment now is open source while factually correct, only 10 to 12% of that code is in use in your environment. So if I could really kind of focus on the code that I use, making sure that it's well maintained, well managed, well secured. If I can have programs to remove the parts of the code that I no longer need consistently, like I think we just have to invest in this, right?

This requires development resources and this requires a programs team that defines what is my minimum viable criteria of how I would use open source. And the analogy I give you, just like you don't wanna buy a a used car without a Carfax report, you don't wanna use open source software without an equivalent. And understanding and reporting of, you know, what am I bringing into my environment, Security technical debt, do you think we have out there given, you know, as we shine a light on these issues, we've been building software using open source components for, you know, nearly two decades now, I think, um, are we gonna discover that, you know, we have a massive amount of software out there that is vulnerable to various exploits?

Absolutely. I think, I think it's, uh, you gotta shine the light first and you have to recognize you're gonna end up with a lot of debt. And the way I talk to a lot of, um, you know, industry experts and CISOs is you gotta compartmentalize your tech debt problem, but also not use that as a reason not to look and say, how do I stop the bleeding moving forward?

You gotta imagine running two programs by right here. Decide at some point that here's my new way of doing things. The new way is well-defined policy is code early intervention.

When developers are choosing open source software, I will recommend the right pro libraries and projects to them. I will intervene if they try to bring in something that's outside of my standards. And I'm gonna do this all very, very early.

Uh, you know, think of it this way. If I can intervene with a developer when they're in the dating phase of a library, much easier than when they're married with four kids, deploy deeply in production, telling them that they screwed up. So have a program that looks forward and says, this is how we are gonna govern open source software moving forward.

The tech that that you have, unfortunately there's no silver bullet. You're gonna have a multi-year effort to burn down that technical debt because it requires engineering, uh, prioritization as well as security investment. But the biggest thing, Mike, you can do with tooling and instrumentation for legacy coding and environment already is how do you focus on the 10% of the code that is actually being used?

Not the 90% that is imported by proxy, but not a, not an exploit. And that is a hard conversation. Most people have software composition analysis tools.

Universally how that market scans is they just look at what you import. This is why at Indoor Labs, we have spent so much time and so much technical horsepower to figure out how to get to that 10% of the code that is 80, 90% of your risk. And I think if you can solve that, well that's how you go burn down like a 10 year roadmap of technical debt into one or two years.

Do you think at some point that AI may save us from ourselves here? It seems like there's some use cases here for some AI that might help. Yeah.

Wouldn't that be nice? There are, look, don't get me wrong, and again, it's the, it's the new world order or the old old world order. Look for your old applications.

You have a library that you haven't updated in seven years. AI can't help you. 'cause there's not a point of like pushing a button, creating a pull request.

That's easy. It is, you just don't know the complexity of your existing software environment, your test coverage, what'll break, what legacy versions your customers are using and so on and so forth. Where we found a great use case for AI was actually to help developers select better open source software.

So when I was talking about that speed dating phase of libraries, well instead of a developer going to Google and saying, what should I use for like a web server in Golang and then bouncing to Stack Overflow, then getting to GitHub expected to do their own review of a library and determine if they should or shouldn't use it. We've automated that whole thing. That's a great example.

You know, we call this a droid G P T product, which is the power of analysis with a recommendation engine through the advances of ai such that developer can just come and say, you know, what should I use for use case X? Like gimme your technical requirement and allow this system, this intelligence system to tell you what are the best alternatives to X that are not just solving your technical problem, but well-written code, well-documented code that is secure by default. So that's kinda how, you know, we, we are applying AI in, in, uh, some of these areas.

We have been talking about DevSecOps for a very long time now. And what's your sense of how much progress are we making here? I mean, 'cause we still seem to be talking about the fundamentals and in your experience, what's the issues?

I think the number one issue, Mike, is what I call the developer productivity tax of this whole ship. Left movement is way too high. Meaning today, the way all of the industry works, we are giving far too many alerts to noise to our developers and, and forgetting that they're not compensated and measured on how much security debt they fixed, they're compensated on how many features are shipping.

So there's this constant tension there and most security tools that end up spitting out 80, 90% false positive, especially as it relates to application security testing in DevSecOps. So that's one problem. The second problem is in spirit, it is great to say, I wanna do things in the left, but you talk to an average enterprise, they're now running 15 or 17 security tools in the pipeline.

Well, who's having to do the work? You know, the ss r e teams are constantly taxed with putting a new tool. Okay, take this out, update this.

It's a lot. So consolidation is the need of the hour, especially as you look at like, like, uh, inspecting pipelines because you know what? You are gonna get 20 minutes to run any tool you want in your pipeline.

If you try to run 15 tools synchronously, it's not gonna happen, right? And so the, the developers will immediately dismiss you, kick your tools out. And so the developer productivity tax is all of this unnecessary busy work that is being posed in the spirit of shifting left.

And reality is, you know, the developers are there to, yes, they should understand their, uh, security topics and concepts better. They should understand some elements of threat modeling, but you can't just suddenly hold them guilty until they prove innocence on all of these issues. So I think the, the thing we're missing is better collaboration, uh, earlier intervention in some cases, certainly much more accuracy from tools that are being run in these environments.

So developers don't become dismissive of the results. Speaking of dismissive, what is the role of cybersecurity gonna be in these application security scenarios? I mean, for years they have compiled these long spreadsheets of vulnerabilities and shipped them over the wall and developers have occasionally hunted them down only to find out that they're not actually using the code that was in that list.

So, um, it feels like there's a disconnect in their relationship. Is that gonna get any better? We hope so.

I mean, that's kind of the mission behind indoor labs is to remove and will eliminate. It is hard, but minimize a lot of the developer productivity tax. But Mike, here's where, you know, regulation, like what we started out discussing will help.

Okay, so you know, now there's a mandate for software bill of materials. Great. Can I just ship an SBO m today as things stand in my industry?

Yes. But as soon as the receiving party scans your SBO against their favorite tool, they're gonna find that you have 10,000 vulnerabilities that you haven't fixed. They're gonna come back to you with a spreadsheet of questions.

And so we have to get better, more proactive and articulate. And this is where I love the vex, right? The N T I A that created the SBO m also created the vulnerability exploitability exchange format, which is ideally they call it a companion doc.

I fundamentally believe in SS OMM without a vex is like peanut butter without jelly because we've gotta share context about why are we not fixing certain things. What did we find in our code that led us to believe that this is not a risk? How do you communicate that outside of your Jira tickets?

Because now you have to communicate those exceptions with the outside world. And so two things. One, vex becomes important as a means of communicating why you're not fixing certain vulnerabilities with the outside world.

But number two, you're gonna push your vendors and force your tools to be much more accurate, which is great because Mike, so far a lot of these tools were run for compliance check boxes. But today, when you have to expose this information to your potential customers or the US government, you can't just send abstract reports anymore. You have to be very crisp and articulate which forces vendors in this space to be more articulated with their detection capabilities.

And by the way, this goes back to what I mentioned earlier, and I especially coming back to open source software, intervening at the marriage stage is too late. I gotta tell my developers when they're bringing in a new library tomorrow morning why they can or cannot bring it in. And I need to have the confidence that it's pre-vetted.

But guess what? With 47 million open source libraries and versions, I can't do this manually. So I need a high degree of automation that's continuing to vet new open source software, new releases and tell me through this early intervention program what I should or should not be bringing in in the first place.

That's how you reduce technical debt because you start clean, it'll help you stay clean, right? You start without any controls and governance frameworks, you're gonna have a lot of tech debt to work with. I feel like, you know, just about everybody and his brother kinda wants an sbo, but very few have figured out how to operationalize it in a way that allows them to take that information and have any kind of meaningful results.

So where are we on that curve? I mean it seems like we need some sort of ss om automation framework, but I'm not quite clear what that looked like. Yeah, there's, there's a whole host of problems, right?

I, that's why I say the trains left the station, but a lot of work has to be done. Number one. You know, we ran an analysis and research where we took three of the most popular open source SBU generation tools, ran it against the same code and the three of them generated three different result sets.

So accuracy is a huge problem. What do I believe? Is it complete?

Where was it run in the pipeline? So that's one problem from a generation standpoint. The second piece is with the velocity of continuous releases, like how frequently am I sharing these SBOs with who, you know, some organizations believe SBO M contains the secret sauce their ip, and maybe if it gets to the hacker's hands, it's giving them unnecessary information.

I don't know if security through security really works, so I'm not on that same page, but industry believes that. But the biggest concern is the people that are demanding these SBOs do not know what to do with them. So we talk to so many organizations, I say, okay, what are you gonna do with the sbo?

What do you do once you receive it? Well, the answers vary from I phone a friend in my AppSec organization and ask them if they can run a quick scan to, in most cases my, I just file it in a SharePoint folder. If I need it later, at least I'll have it.

So you're right, there's a lot of conversation in the industry around SBU production and generation. Not a lot of om consumption that is an under invested area. We at Pendo Labs are certainly spending a lot of time on that, you know, but as an industry, yes, we need to understand what are we gonna do with this information?

It is a lot, right? It's an 800 page j o came document, what do you do? How do you make sense out of that?

How do I know what are the risks? I really care about what, why not? And and how does that closed loop process of then kind of the customer going to the vendor saying, I have 17 questions and here's a spreadsheet, vendor reacting, you know, getting the context from their development team.

It just requires a high degree of automation that, you know, folks like us and others in the community are trying to build. But customers need to start understanding that that next set of pain points that they will encounter was the gas bombs. Last question.

What's your sense of the bad guys out there and their sophistication for these kinds of attacks? 'cause you know why we saw the log four J shell attack, it's not clear to me. I saw a lot of widespread, uh, compromise.

So, and frankly this bad guys have a lot easier ways of compromising things. So what's your sense of what the real level of threat here is? Yeah, Mike, there's two sets of problems here, right?

One, I think rightly so, the US government is putting a lot of attention on this. They call it a, you know, national security issue with open source usage. There's a bipartisan bill now in the Senate for securing open source act.

Um, so these are all good things. Now what's the reality of like situation on the ground? There's vulnerabilities and then there's the unknown attack factors.

What are kind of being constituted as a supply chain attacks vulnerabilities, you're right. Like only like less than 10% of the vulnerabilities are actually exploited in the wild. So one huge metric that we think organizations need to follow for their vulnerability programs to be successful beyond just C V S SS and CV scores is actual exploitability data.

There's a great open, you know, an open source kinda consortium providing this called E P ss SS exploit predictability scoring system. We think that's gotta be critical as a mechanism to prioritize your vulnerability reports internally. Now, beyond vulnerabilities, we are starting to see, you know, various different attack vectors, but this is where the malicious people are putting packages with malicious code out on the internet.

The problem is your C V E scanners don't cache these, right? 'cause the C V E scanners are typically catching well-intentioned developers made a coding error. It's now understood and documented and they're working with the community to fix it.

These are malicious packages. We're starting to see tens of thousands of them every month being pushed into N P M into pi pi. You know, they're following things like name, confusion, attacks, and a lot of other ways to do it.

com and you'll find a risk explorer, which gives you an interactive way to understand these new risks classified with actual examples of real attacks. Uh, but, but Mike, that's the problem. As an industry, we have to evolve beyond vulnerability and license risks to look at open source more holistically and beyond malware.

The examples I just gave, you know, even maintenance of code. Like is the code well maintained? Is it now dormant?

Have the maintainers gone away? You know, so many times people are advertising in the wild saying, Hey, I'm giving up this project. Is there anybody to who wants to take this project on?

And this project has hundreds of millions of downloads. If I'm a malicious actor, I take on control of that project, I can push you an update with all sorts of things. So it's important to grasp these different attack vectors and think about your roadmap internally to secure yourself against these attacks.

Folks, while you heard it here, there's a fellow out there, I think his name is Murphy. And you know, if you decide that you're gonna not do something, the joke's gonna be on you eventually. 'cause the odds are somebody's gonna find that vulnerability and exploit it.

It's just a question of how much damage they're gonna inflict and what you're gonna do to remediate it afterwards. Vern, thanks for being on the show. Yeah, thanks so much, Mike.

All right. Thank you for all watching the latest show. And back to you guys in the studio.