Software Supply Chains in the Age of Open Source – Javed Hasan, Lineaje

This is Textron tv. Hey guys, thanks for the throw. We're here with Yaba Hassan, who is c e o for Lineage, and we're talking about how far left to ship responsibility for application security these days in a world where we're trying to lock down our software supply chain.

Ya welcome to show. Thank you Mike. Excited to be here.

There's a lot of debate about this issue and it's been going on for a while. I mean, on the one hand we've been saying, well, we should shift left to developers and then they complain about the cognitive load. And then there's another debate about how far left should we shift all together and just how far can we take this beyond the traditional security play.

What's your sense of what's going on and how do we kind of get at this issue? So I mean, I I the, the way we like to phrase this is there is a shift left where software for enterprises and organizations is built. The challenge we increasingly have is 70 to 90% of that now comes from open source and other third parties.

So what's happening now is that 70 to 90% of the software that organizations are shipping now comes from what we call left of shift left. So shift left in our view, you know, is giving incomplete visibility, incomplete security, and incomplete control on where the, you know, on the software that is being shipped. But the majority of it, like, like you said, is being, is now left of shift left.

So increasingly you are seeing the emergence of the software supply chain movement is, is what I would say where we start giving organizations visibility into left of the left. To your point, we have dependencies upon dependencies upon dependencies, and a lot of times I'm not gonna be in a position to fix something, even though I know it's an issue because it may have been created by some sort of downstream maintainer for me. So how do we build an effective supply chain where not everybody in the community has the ability to, uh, have complete control over every component that they're using at the moment that some vulnerability might be discovered.

And I got maybe 10 minutes worth of warning about. Yeah, so, so if you, you know that that's a first, that's a great question and a great insight. So now you sort of think about it, and again, you know, I work for Lineage and so, so our research reveals the following, that 95% of a weakness in, so in, in shipping software now comes from the software supply chain or the left of shift left and only about 5% is, uh, is from the code of company rights.

So, you know, you can see the, the challenges moving to left of shift left as opposed to right and AppSec security tools by and large focus on shift left security. So that opens a big gap. And, and, and, and the billion dollar question in my view is how do you now secure left of shift left?

So, you know, one way of sort of now dissecting it a little bit that now like you said, that the dependencies are multi-level deep in, in fact, now we are seeing, you know, dependencies being 20 levels deep. And now if you have availability that is, let's say in a direct first level dependency, maybe you can patch it to an independent patch that you can get. So your developers can patch some parts of open source, the one that they can see easily and that has independent patches.

The rest they can't really patch or fix. So increasingly, I mean the, the, the assumption that developers can actually fix 80% or 90% of the issues that you are seeing in software, the answer is they cannot. I will give you one more data point that we are seeing that most organizations have developers that deal in five or six languages, you know, they have a certain skillset when we are pulling in open source, we are seeing things like 29 languages or 30 languages of dependencies being pulled in.

So in and from, even from a skillset perspective and organizations, developers don't necessarily have the skills to go patch and fix, you know, vulnerabilities and weaknesses in their dependencies. So that's the big challenge and that's why we started Lineage to sort that problem out and we can get into how we think that can be solved. Well, let's go there right now because it is the billion dollar question as you put it.

And people are struggling with how do I kind of build a software supply chain that goes beyond just creating a software bill of materials that tells me, you know, where my vulnerabilities are because I can't act on it. So how do I turn all these SBOs into something that is actionable intelligence for a software supply chain that I can actually manage? So, you know, if you, if you now think of it in all of these situations, and we have been, you and I have been through this many, many times, the first thing step really is, you know, do you have the entire visibility that you need so you know the problems you have to fix?

And the second one is you pick each problem at a time and you fix it. So if you sort of think of it as those two steps, so let me take you down, that's very specific path. So if you start, if you start now looking at dependencies, and let's assume that there are deep dependencies on each one of them go let's 10 to 20 levels deep, we have already attest or certify that we can't go fix a developers can't go fix that, that that deep wants.

So the first thing that we do in, in terms of lineage, we actually divide it into where is the source of the dependency coming? And so is it open source, is it third party, right? Or is it your own code?

And the ways you want to fix it are different. So third parties, you can turn around to them and say, hey, 17 levels reap you have vulnerability, please fix it. And so when you are buying software from someone, you can turn on them and make them responsible for the fix.

That's one part, even it's a component in your software, right? The second thing really is that it's a true open source dependence. Now what we do is we look at it and we've essentially divide it into buckets.

So we know, for example, it's a well maintained open source project. So in general you can predict that there's a certain time duration in which they will fix it. So one of the big, so, so now you start dividing the problem your dependencies into, oh, these are well maintained.

These ones have not been maintained for two years. So now if you have a dependency on a, you know, on, on a project that has not been updated for two years and now you have a critical vulnerability or some other weakness in that code, the chances they will actually fix it are look or or non-existent by and large. So you need to know that, that you have a weakness and you can't depend on those guys to fix it or you have a weakness and you can depend on that to fix it.

And the third bucket, of course, is in the middle, which is they fix some, they don't fix others, right? So then you have the ability to push what is, you know, to log a issue with the dev with that project and try and drive it through. So ultimately, like most of supply chain problems within the physical world, what this boils down to is managing first, being careful in selecting what is in your software supply chain in your dependencies.

First, doing it with full knowledge, not selecting projects that are not maintained well, selecting projects that are maintained well and then pushing the right issues downstream or you know, in this case upstream to them and then investing there to, to fix it. Now the last point I would make to you, my case that we are seeing a, a emergence of this movement that is being called inner sourcing. So now I mentioned that you have dependencies that are, uh, not well maintained, but you really need them.

So you can't go replace them with something else. We are just critical to you. So inner sourcing refers to this vulnerability of taking open source, bringing it in house, funding it like that.

You will fix it, you will manage it. Is it continuously so that your risks are minimized? So we are seeing large organizations starting to waste in inner sourcing, creating inner sourcing teams that manage specific open source buckets, right?

That they are really dependent on. But we are sort of seeing the emergence of solutions around that. So we build something we call lineage AI to help organizations go down this path.

How do I also address the following scenario? Because sometimes I will talk to the maintainers and they'll tell me, yeah, I got that request, but we fixed that bug three releases earlier and it turned out that the organization didn't wanna upgrade that software to the latest version because they were afraid the application would break. And it seems like there's a lot of vulnerabilities running in our production environments that can be remediated that people don't just because of fear of breaking the application.

So how do I kind of, in addition to shifting left kind of reach right a little bit and close this loop. So, uh, it it's, it's an amazing question and, uh, Mike, and so we, you know, do we call this the compatibility problem? So what's happening is we have seen AppSec tools go down this exact path where they find vulnerability tele developer to fix it.

Developer fixes it by going from V one to V two of a component. The problem is that there are, let's say SBO M has a thousand components and now you change one from version one to version two, and version two may or may not be compatible with the other 9 99. Now if you actually go to version two, it may break 50 other components.

So now you have this cascading problem. Now the issue that we are starting to see here is that this, let's call it patching on demand or updating on demand has a high cost for, for organizations. In fact, the compatibility testing overload, the tax that is being paid is about 40% of all maintenance resources.

So you've fixed, right? So what we are now doing with the AppSec tools is really creating this bottleneck in the, on the right, if you will, where developers willy-nilly are upgrading little things, the compatibility is not taken care of and is not, right. And so now you are creating these testing bottlenecks where organizations are afraid to take patches because they're, they don't know what the compatibility tax is going to be.

So one of the things we did with linear AI is we actually built, so linear AI is overall AI that pulls together what we call a bunch of capabilities called bomb bots, bill of material bots. So they work with SBOs to essentially fig, you know, solve all of these problems in a single interaction. So, so one of the bomb bots is something we call a compatibility bomb bot.

What a compatibility bomb bot can do is it'll look at your entire sbo, right? We start, it looks at your sbo, you have a thousand components. We know because you have tremendous amounts of data.

That version two of this works with version seven of that and version 19 of that. So in general, what we can, what the compatibility bomb bot can do is can look at all your vulnerabilities and all your weaknesses, look at what versions they have been fixed in, of various components, and build you the best compatibility matrix for you to say, look, this particular vulnerability to take a very specific example is fixed in version two, version three, version four and version five of this particular package. Now, which one should you take?

The default for a developer is I will take the latest, the problem is the latest to version five may not be compatible with the other 9 99, but it could very well be that version three, which fixes the vulnerability and it's also compatible with the rice, right? So you can, we can now create, let's say, planning SBOs from the actual SS o. That's it.

If you, you go from version one to this planning, ss om type one, which is the least effort, most security, uh, impact. Or you can go to V two or version, you know, type two of this, which is high effort, the most secure, right? And, but by the way, 17 of these, uh, the of, of, of these packages will break in compatibility.

So you have to the cost of testing that you'll have to incur. But, but in general, what we are really going after here is to say, look, maintenance done thoughtfully and better will accelerate security. Mm-hmm.

And so we are trying to help the bombards are trying to help, AI essentially is trying to optimize exact all this exact problem across the left of shift left and shift left. We see a lot of regulations that are starting to be proposed out there by various governments is are they gonna essentially force this issue? Because I feel like we haven't really paid a lot of attention to managing our software supply chains, or at least not as much as we should have over the last decade or so.

And are we gonna have a, a reckoning essentially as those regulations come down the pike and people are gonna have to take a hard look at how they're actually managing this process. I, I, I, I, you know, my personal view is I think we are, you know, relatively rapidly coming to that point of reckoning. I think what the government seem to be saying is that in every other, in every other vertical where if I have a car, I can hold the car manufacturer, uh, responsible if something is wrong with the car.

And it's rarely the, the, you know, if something is wrong from a design perspective, software is one of the places where a manufacturing software or a software that can become the vector of an attack that a customer loses data on actually the customer more because they now are, you know, they are subject to breach laws and all of that as opposed to the vendor who provided the weak software. So increasingly I think we are seeing this movement saying there is a responsibility on companies that make money by shipping software, that there is actually a higher, uh, accountability that will be needed. And I think we are seeing some of those significant legislations come through in Europe and so on, and I think they have global implications and I think the US potentially will end up there as well.

And I think software developers have to be more accountable. I think the debate continues to be how much is open source accountable. Mm-hmm.

Right? And there it's, it's a much harder discussion because open source ain't making money, right? That may not have the resources to fix everything.

And they are essentially subsidizing large software development at this moment. So large software develop large professional software development needs to step in, but first of all, needs the tools like linear AI to help them fix it. Do you think the organizations that consume open source software have some sort of obligation to help fix some of these issues in these projects that they're using software that are encountering these issues?

Because to your earlier point, not all open source projects are created equal. Some have a lot of resources and some turn out to be like log four J where it's basically two or three people hanging out on a weekend. So, um, how do we kinda, you know, figure out what the responsibility is of the corporate companies that are using this software, if any?

I, I, I think there's significant responsibility on the corporate companies. I think in no other industry would you let an engineer go get a part from a local garage and put it in the next B M W, it just doesn't happen, right? There's a certain way of selecting your suppliers, and I think most of the companies have let individual developers, and I would say, you know, the wrong, you know, not necessarily the wrong people, but, but without select office suppliers, whether open source, I mean specifically open source without any due diligence, just because it, it has a certain set of features, right?

And whether it satisfies, you know, is it the long-term, uh, long-term dependency they wanted or not, but that, that sex has been lacking and we are seeing enterprises turn around and start paying micro attention to what OMI source they're dependent on. How, how will it play out over the longer term? And I do think that whether they subsidize it or not, there is a responsibility if you're going to consume a piece of software for you to then help whether in people or in summer or in management or in some other way to make sure it remains secure and usable for your customers.

So ultimately, what's your best advice to folks then as we kind of look at this, it's a little overwhelming on a certain level. So how do I get started? Where do I begin?

So I mean, one of the ways we recommend you start is first, first step really is know what your dependencies are. So what's in your software and a, a software bill of materials like we create is a great starting point. Segment it it into what's your dependencies and what's built in-house and what's third party?

So your open source, third party, and private. Then we can assess all open source projects, for example, for you tell you where the, how good or how bad they are, how risky they are. And then comes the path, like I said, of managing your source software supply chain.

And I think in West, I see no path but one that companies need to start investing as much as in any other industry in ensuring that the software supply chain is well managed and well maintained and they will have to throw resources at it and they'll have to build that in, build that into the cost structure or the software they deliver. All right folks, you heard it here. It all starts as usual with the first step of a long journey, but the longer you ignore it, the harder it's gonna get ya.

Thanks for being on the show. Thank you, Mike. It was a pleasure.

All right. And back to you guys in the studio.