Jim Zemlin, Linux Foundation | DevOps Connect 2022

My name is Jim Zellman. I'm the executive director of the Linux Foundation. How many people here know the Linux Foundation mostly.

All right. Wow, that's that's good to know. You know today.

I'm going to talk a lot about what's wrong with open source, you know, I think people realize that open source as of late particularly in the last six months has had some really high profile security issues and has had some attacks, you know on multiple vectors, you know log Forge was probably the most significant but there's been a series of escalations and supply chain attacks in the open source community that are disturbing. So I'm going to talk a little bit about what those are but before that I want to remind people about how important have been source is and how essential it is to all of our Are collective Innovation and sustainability of really the entire tech industry one of the things I like to talk about when describing the Linux foundation and what we do is take a little trip down memory lane to when I first started this job, which was almost 20 years ago. I that year met my now wife on a blind date and when she asked me what I did for a living I said, well, you know, I'm working on this open source stuff.

I work at this Foundation. It's called the Linux foundation. And the look of disappointment was just palpable and in the intervening period you know in the 20 years that I've been working at the Linux Foundation a Linux is essentially become the most dominant platform in modern Computing Yeah by our estimates 90% of all modern Computing workloads run on Linux.

You know, when you look at the fact that Android uses the Linux kernel and the mobile space when you look at High performance Computing which is essentially a hundred percent market share for Linux supercomputing. Same thing embedded systems vast majority market share Enterprise Computing. It's the largest market share and so on so forth.

Sometimes it's hard to remember just how popular Linux is because you know, the desktop has been an area where it's never really broken through but that really is a very small percentage these days of modern Computing workloads. So this is essentially a ubiquitous building block for all modern technology products and services at least the vast majority of them and the Linux Foundation has grown with that today. We have over 2300 organizations that are a member of our organization from all over the world.

We have hundreds of thousands of developers that work in our communities every single day on the most critical open source projects in the world. This is the largest shared technology in In the history of computing and we're super proud to be home to this critical part of the tech sector, you know, what's even more insane is how fast our organization continues to grow in terms of new projects that really impact the tech industry and the number of organizations that participate in our world. We're adding about one to two new members every single day and have been for several years so clearly open source.

It's important something's going on. You know, our name is the Linux Foundation, but one other thing that's important to understand about organization that I think most people don't know is that we're much more than Linux. We're the world's largest certificate Authority.

Let's encrypt is a part of our organization. We're home to you know, critical projects like kubernetes in the Telecommunications sector are open network automation platform orchestrates and manages. Vast majority of the world's mobile providers production workloads in the film industry.

We're home to a partnership with the motion picture Academy where we're working with all major film studios to open source, the building blocks for modern digital effects. So if you go and see Marvel movie or Star Wars movie the software that's being used to create those digital effects is all open source. It took us several years to work with the industry to open source those building blocks and you know, they're postulate was you know, Disney and Pixar and Lucas Films are not software companies their storytellers and by collaborating on the software that they use to build these incredible films.

They could produce richer more beautiful effects and therefore more compelling stories and this story just keeps repeating itself in the energy sector. We have a project called LF energy we're working. With grid operators in Europe and around the world to modernize the distribution of energy and make it more efficient so that we can reduce the impact of the energy sector on global warming.

In the hardware sector, we have projects like Risk 5, which is really the fastest growing semiconductor instruction set in the world right now, and it's really poised to become a dominant force in the hardware semiconductor industry. So I could keep going on and on and on I think the important thing to remember about open source, is there sort of a counterintuitive way of thinking about it and that'll get us into the security part, which is that you know, there are I can't remember what the count is most recently on GitHub, but it's in the, you know tens of millions, you know, 30 40 million open source projects out there in the world and you always think of Open Source as this huge diaspore of millions of projects and millions and millions of Developers. But there's actually a sort of a power law to the open source projects that really are critical to modern technology infrastructure.

And those that we really need to focus on from a security perspective. And and that number of projects is actually much much smaller and more like in the 10,000 range in terms of core operating system components core application libraries and different vertical Industries and then, you know sort of the top dependencies and the different language Frameworks that we all think of whether it's in, you know, JS or Ruby or Java or so forth. It's really not Millions.

It's more like thousands and the number of developers that actually maintain those code bases is also not in the millions. It's much more in the thousands. The Linux Foundation has done quite a bit of analysis on these critical open source projects and partnership with Harvard University.

If you go to our website you can just Look at these reports on software criticality. one of the things that's important when we think about these great shared software platforms that we all depend on to build the products and services that runs, you know, all of the different companies that are represented here today is that at the Linux foundation in general in open source, you're kind of shooting for this positive feedback loop in terms of making sure that an open source project is healthy and and secure the idea here is that you want to find first a project Market fit, you know, like if if the open source code itself doesn't solve a significant problem, you know, it's just obviously not going to be used it won't be well maintained because No One's Gonna really care, but when you find a Linux kernel or a kubernetes project or a nodejs these create tremendous value and you start to sort of have this pylon effective people really not only utilizing it to tremendous value, but also creating products and services based on that software and when organizations create a product using the Linux kernel or kubernetes what happens in that Process is a fine vulnerabilities. They make improvements to the code base based on their particular requirements a good example of this is in the Linux kernel when Android really started taking off one of the things that developers from that ecosystem contributed back to the kernel project was power efficiency code, you know essentially ways to extend battery life for mobile devices within the kernel construct and that had a unintended benefit of really reducing the number one cost in major data centers, which is power and cooling.

So when you have Different organizations that are consuming these really large-scale projects and productizing them you're getting this tremendous investment in the engineering and maintenance of that project and this cross-pollination and unintended and sometimes Innovation that comes from that and of course, you know, a lot of you work at organizations that create technology products and services and you make money from that and you know folks in open source are often, you know, seeing as like not caring about money and you know, when you talk to a developer like a leanestorevalds or maintainer of kubernetes, I think there's some truth to that money is not their number one motivation in terms of participating in these projects. They really see it as getting their job done being a part of something bigger than themselves mastering their craft the goal of technical excellent Excellence, but the organizations that they work for certainly are here to make money nothing wrong with that and some of that money when companies are making money building products and services with open source gets invested back into the open source projects not most cases through Direct Cash contributions to a developer within the project. It's more just employing and paying a developer to work on that project as part of their job.

This is something where we did a recent survey of developers around the industry that are critical maintainers and open source projects. And the number one requests that those maintainers ask for is getting their company not to contribute money to and open source project, but just giving them more time in their work day to spend on maintaining these open source projects that that's something that they really care about and when you get that a great project that create that is used in products and services that those products and services generate profits you get a reinvestment back in the project and you get this sustainable positive feedback loop and for the greatest open source projects and we can talk about problems that they have around security that aren't being covered by this positive feedback loop, but in general these are pretty darn Healthy Communities that produce sustainable reliable code that everybody depends upon the reason talking so much about this is some of the most high profile cyber security incidents that we've seen as it relates to open source happen when this feedback loop fails And the tricky thing and what I like to talk about today, is that for all good open source projects. They tend to look like this.

They tend to have this positive feedback loop. They tend to have full-time people who work on these projects for all sort of dysfunctional or broken open source projects, you know, if you go back in history and open SSL or you know, many of the highest struts or whatever it might be the there really is no pattern as to why they've broken down or why a lot of security problems that happen there's some patterns but it's actually very total story-esque each miserable open source project tends to be miserable in its own unique way and that's been one of the challenges around raising the collective Baseline for cyber security around open source projects. But let there be no doubt that open source is indeed a fundamental building block of Technology.

One of the things that I talked to Regulators about or folks in security often is hey, if this open source stuff has security issues can we get rid of it? And you know, just we'll use some different thing or some theoretical thing out there and the really important thing for everyone to understand in 2022. Is that the open source Bell cannot be on wrong like to rewrite the Linux kernel code would take billions of dollars of effort and just I can even imagine how long it would take to rewrite all the device drivers in that ecosystem retool the entire industry and you know, this goes on and on up and down the stack.

So, you know, well, there are things we need to do to improve the security Baseline for open source. It's really important. Understand that a this is an irreversible set of fundamental reusable building blocks for all major technology products and services and that is not going to change 90% of the code in most technology products and services is open source.

And the reason is it's high quality software. It accelerates time to Market by not having to build your own operating system in middleware framework and cloud computing components and so forth and what organizations long ago figured out is if they just focus on the 10% of their code that their customers really care about that's really unique to what value they're bringing to the market. They're able to produce this software much more efficiently.

You know, it's not only just a smart strategy. I don't think you could do it any other way. I mean, there's just more lines of code in a modern automobile these days then in an F-18 fighter jet.

It's just they're simply too much code that needs to be written for any modern technology product or service for any single developer or developer organization or company to write on their own. That Bell has definitely you know, it cannot be on wrong. The important thing to remember is that there is a critical set of these different open source components and in some of those components there are security problems, and that's the problem that we really need to address now.

And the reason for that is software supply chain security breaches are significantly up. We're really seeing a and you know RSA is I don't really need to explain it too much here a Confluence of a much more sophisticated set of cyber attackers. We're definitely seeing an escalation of cyber conflict even at the nation state level, which is disturbing and you know, we're just seeing more and more components that are creating weaknesses every day, and we need to do something about it.

And that's really what the Linux Foundation through our open source security Foundation is looking to dress open source is irreversibly a part of modern technology. We know that we need to raise the security Baseline across this supply chain. It's obvious to everyone that attacks are on the upswing and that the cost of these attacks are tremendous how many people here were personally involved in remediating log Forge issues.

Was it a cheap thing for your organization? I mean I can't I can't imagine and it's gonna haunt us for like years, right? You know, like that's the even worse part about it.

But I mean, these are not cheap to remediate this stuff is a super expensive in a rears to deal with and not only are we getting more of these types of attack vectors, but now the government is really demanding action, you know, I mean, I think folks recently saw the FTC is warning folks that there will be fines and potential legal action for organizations that are not remediating some of these critical vulnerabilities in particular in light of long Forge. So this is kind of some of the bad news about the attacks and what we need to deal with but there is some good news here. One of the things when we were working with Biden ministration with Ann newberger and easterly Chris English and folks from industry, you know, Cisco's from all the major tech companies Amazon Google Microsoft Etc.

Is we talked about? A series of programs that we could do to raise our collective security Baseline and open source, and we threw around this idea of calling you to moonshot. and I just thought that was such a terrible name because you know moonshot involves some kind of like incredibly Innovative new thing right where you're really striving to the boundaries of innovation in order to you know, bring a person to a moon or something like that that really dealing with here in terms of what needs to be done to collectively raise our security Baseline as it relates to open source.

I mean most of these problems are actually fairly obvious and I'm going to take you through this and most of the solutions Aren't actually all the all due respect to cryptographers and stuff. It's not all that interesting. It's not all that Innovative like package signing this freaking 2022 and like it's not particularly Innovative.

We just need to have the will to go do this stuff, right you reproducibility of software builds, you know, a simple trainings in application security for developers. These are not super Innovative things. They're just stuff that needs to be done.

We need to have a collective mobilization to get this done. The other sort of bad news about this is when you're talking about a mobilization plan for the open source Community. It's a little bit different plan than for a single organization the way to think about it.

Is this how many people remember back and I think it was 2003 2004 when Bill Gates wrote a letter one of his open letters on security stopping all software production at Microsoft, you know, essentially saying, hey, we're gonna refactor our code we're gonna go test this stuff. We're gonna do, you know a whole, you know, cyber trust initiative. So how many people remember this letter way back then where NT was having all these security issues Gates just said time out, we're gonna address this and the implication was if you're an engineer at Microsoft and you're not involved in code reviews and testing and security practices.

I assume you're probably gonna get Appeared but here's the rub when it comes to open source. We can't go fire a million developers who work in an organic self-forming collaborative development model. What we have to do is create a culture and a consensus of security just like there's a culture and consensus of sharing, you know open source.

We need to create a similar culture and consensus of security and if we do that in a few simple things, I'm going to show you today. I think we can massively race our collective security Baseline make it a lot harder particular for rogue actors and individual actors to exploit the open source supply chain. I think most people in this audience understand this but one of the things when we talk about supply chain security and open source security that's important to understand is sort of how code flows in the open source world and in software production in general.

I mean code comes from creative developers who work in University Control Systems, whether to get GitHub get Lab at lasting you name it where they're actually writing code, right? They take that code that code is built using a build system oftentimes developers will combine a ton of different libraries and packages. So this is sort of where package managers come in where you've got a whole bunch of reusable, very useful software components that you can basically build into whatever you're creating and what this does is it creates these dependencies where I'm pulling in all these different Components into the software that I'm building a package that up I shift it off to my consumer.

That's sort of how all this stuff flows the major package managers are pretty obvious to everyone. It's you know, npm, you know Maven Central for Java ruby gems, you know, these are where all these dependencies come from. You sorry you sort of core operating system middleware components that are critical in different Industries.

You might have networking components in the telecommunication Center. They're in that are critical and then you have based on what language particular developers using or what framework they're using all these dependents this that come in. So for any modern technology artifact you have thousands of these different components that come in.

and as code flows down this chain, there are Areas, where we have sort of endemic weakness in the chain, right? And this is where attackers can come in and take advantage of that, you know, if a developer want, you know one simple way is, you know, they maybe bypass a code review or nobody's there to review any code and they never really learned good application security practices. So they just kind of you know, unintentionally wrote crappy code that has a vulnerability in it and that subjects all of us to a lot of risks.

From you know, I love the folks at the Apache software foundation. And you know, if you we did some retrospective history here on log Forge, you know, there's an argument that if some of the developers who accepted patches that resulted in the vulnerability had taken some application security courses, they would have known that what they were doing probably wasn't a good idea. So this is an example of where and of course hindsight being 2020 and you know, these folks were under resourced but in retrospects if we had gotten ahead of this and gone out and offered what we knew were critical project maintainers application security courses so that they could just write higher quality more secure code that attack Vector would be de-risk considerably and you know as we go down whether it's you know in a package manager like package squatting using someone's names.

Just talking to someone today is the maintainer was a million downloads a month for a particular package. I didn't hear which framework was it in. Is it a job?

So it's an npm. It's a node project, you know one maintainer. Not a lot of code review it sounds like but what you talk about it later, but you know, if someone abandons those projects somebody else who's a bad actor comes in.

These are all different attack vectors in the supply chain, right? And that's really in each of these different weak points, whether it's a package manager, whether it's in the Persian control system, whether it's in build systems, whether it's in reproducibility, we need to go and do some software development and do some collective diligence to really Shore up these weaknesses. And this is really what the open source security Foundation is focused on doing and what we have here is just a whole bunch of different projects that over the last year.

We've raised resources for from some of the largest and some of the most Innovative security companies in the world both through funds to directly underwrite the development of some These programs but more importantly a lot of the big tech companies are seconding their staff who have the real expertise that's needed to solve a lot of these issues and build a lot of these systems. To go out and do just that and while this is a little bit confusing and it sort of shows you where we're working across that supply chain. What I thought I would do is go over a 10-point plan that we presented recently at a meeting with the vitaministration and folks from the White House and Newburgh and others in Washington on really Taking meaningful action and securing the open source supply chain.

And so there's three major goals to each of these. The first goal is really starting with securing open source software production itself. And you know, as I stated if I look back at a lot of the security vulnerabilities and big high-profile open source problems that we've had there is a very very strong argument that if a lot of those developers had been offered free secure coding practice courses many of them and we've talked to a lot of them would have taken those and probably would have written more secure code and definitely in several cases would tell us privately like, hey, I probably wouldn't have accepted that patch had I known what I know.

Now how many folks how many Guests you have of how many universities in just in the united states require a secure coding class in order to graduate from a computer science. degree program it's zero. I actually didn't know that I thought it was kind of be one right it's zero.

And that's a pretty stunning fact in a world where cyber conflict is at the fever pitch that it's at today where software security issues are so high. We're just bugs and software are making all of us collectively miserable. And so one of the first things in this plan that we're doing is creating a series of training courses that will be freely available that we will Syndicate through partnering with universities offer free online offer in person learning courses at campuses, you know corporate campuses around the world.

We've already built out a bunch of this course coursework and we've raised roughly 50 million dollars across all these programs a big chunk of which is going to this to get this job done. But when you think of this first stream in terms of just helping developers understand application security. This is not particularly innovative It is a lot of work.

It's just a lot of doing to build the courses get them in front of folks provide incentives for particular developers of critical open source projects to take those courses and then apply that learning to the software they're being developed. But this is the job that's ahead of us. And the good news is that we're already underway through the open source security Foundation.

The second thing that we need to do in order to raise our collective security Baseline is have some kind of situational awareness in particular for these critical open source projects about the health of those projects. And what I mean by that is, you know, if you go and look if just you just do a Google search on Harvard and census to this is a report that we wrote on how you define what critical software packages really are in open source, and the idea here is to focus on that smaller group and really get Awareness on has this code been touched in the last couple of years or not. Has it been abandoned?

Do they have a responsible disclosure policy is there serious test coverage like is there it is what's the mean time to pull requests resolution? Like what are a set of Baseline metrics around secure coding code review testing responsible disclosure vulnerability remediation and so on and so forth if we had a really good reputation system that really doesn't exist today. There's pockets of this around either a few commercial products or a few, you know, sort of leading indicators maybe on GitHub and so forth, but there's no real serious reputation engine if we have that I think people would a make a lot more informed decision about the software that they use in the production systems that they build and be we would do a much better job of getting ahead of problem projects that we're all sort of vulnerable to Knowing early there's something off here.

We got to go fix that. Another thing that we want to do that's not particularly Innovative, but is a ton of work is just accelerate package signing in software releases. Like it was only recently that npm adopted any kind of digital signatures.

Like it's just it's 2022. Like we really need better package signing better reproducibility across all the major package managers. And so one of the things that we're doing at the foundation is we've created projects like Sig store that really provide a more modern architecture that we're offering to all the major package Management Systems in order to provide for this kind of package signing recently.

The kubernetes project adopted Sig store. That is a great start. We've been in talks with Maven Central and others but I mean, it's really sad in 2020 to see the lack of this and as a result, it's just an incredibly straightforward attack Vector for folks because we don't have this.

And then the final thing that we're trying to do to secure open source software in terms of How It's produced is is go and rewrite some significant packages using memory safe languages like rust and you know, we can have a debate about whether rust or go or see you're like what different languages there are but certainly if you look at things like rust it because it's a Memory safe language. It reduces in a major attack Vector in particular in some of the crypto libraries around networking. There's initiatives.

Let's encrypt and at the open source security and Foundation to go and rewrite some of these critical libraries in memory safe languages and again, not all that Innovative. Yes a lot of hard work, but not even all that expensive in relative terms and significantly improves our Collective Baseline. The other thing that we're needing to do is improving vulnerabilities discovering remediation.

I would argue on vulnerability discovery. For somewhat decent at finding bugs and software. I would say that the industry is totally s***** at fixing those vulnerabilities.

Like it's just a real problem. If you're a maintainer even in a very sophisticated project like the Linux kernel just the number of you know bugs that are found. It's very hard for the maintainers to keep up with that and what we probably need here is to a accelerate discovery of new vulnerabilities by maintainers by having better scanning tooling better testing within the different open source projects this may surprise you but there's just a ton of Open Source projects out there that don't really do any kind of vulnerability scanning at all where they really it's just not a thing that's a part of their modern development process.

A lot of these projects are older projects that are on not really on a modern SecOps platform and have an adopted Modern security secure coding practices and one of the things that we're doing is providing a lot of these projects with tools that will help them discover bugs and then equally importantly going and providing expertise and actual time from different technology companies to remediate a lot of those bugs. This efforts in something called the alpha omega projects that we're working on at the open source security Foundation. The Omega side is really looking at the top 200 most critical open source projects and doing essentially manual audit.

js and others a typical software audit of code basis of that ilk ranges anywhere from 50 to 100,000 every time we've done one of these in the colonel or in kubernetes or other critical open source projects. We have found really critical vulnerabilities, and we've been able to remediate them and really collectively raise our Baseline on that. For the longer tale of the sort of the next 10,000 open source projects.

That's where automation comes in where we're working with Major Tool vendors companies like sneak blue bracket Microsoft Google many others to give Collective awareness and alerting of vulnerabilities through automated scanning and then we've even created a slot team of Engineers that will go and try and fix whole classes of bugs in critical software components and in partnership with the maintainers of those open source projects. The other thing that we need to do is when vulnerabilities do happen many open source projects out there. Really don't have a great set of processes to deal with this.

And so one of the ideas that we're working on and is already underway is to establish essentially a volunteer firefighter Corps and like volunteer firefighter isn't really a good description of this. It's more like getting major technology vendors who depend upon this software and it to sort of Pay It Forward by providing engineering expertise when a vulnerability is discovered to get their engineers in there work with maintainers quickly remediate so that the impact of these vulnerabilities is less essentially surge capacity when you have a log Forge or something like that and companies like Google Microsoft Amazon and others are already stepping up with software Engineers to staff these kind of efforts. We haven't had one in practice yet, but it's something that I'm looking forward to seeing an action so that when we do have The inevitable next major vulnerability that these get remediated and are have far less impact on all of us than in previous cases.

Finally, one of the things that we wanted to do is have a global data set where we're sharing vulnerability and essentially data and essentially commoditizing a lot of the vulnerability disclosure and testing results across the industry and what's going to happen. There is two things. We're gonna have more liquidity in data that we all share in terms of responding to software vulnerabilities, but what it will do at the same time is force the security industry to raise the baseline from you know, just scanning code and using, you know, sort of freely available CVS data to send out alerts to really up the game to Predictive Analytics policy engines and so forth.

I think that's a critical thing that we need to do. You see the best security companies raising their Innovation bar that way Better Baseline sharing of data is going to accelerate that effort. It's not only going to help to be able to faster remediate these problems, but it will raise that Innovation Baseline in the security industry itself.

We've seen that same sort of raising of the Innovation Baseline and other open source worlds. This thing is when something does happen get out there quickly. This is something where in last year's executive order.

You saw the mentioned a software bill of materials specifically as a way for people to be aware of what software components. They're using in their technology products and services and in the industry we find they're sort of more, you know, kind of men in terms of end users of technology in particular understanding what software components. They're actually using in their production systems.

The idea here is that we want to use some of the existing software build materials data standards that are out there as PDX is one that we host at the Linux foundation and improve essentially the liquidity of software metadata across the supply chain. This again is not particularly Innovative, but it is a ton of work. Let me give you an example one.

You need to create tools that allow for open source component developers to easily create a sophomore building materials file, right? So you just need automated tooling that allows an open source developer to maintain an spdx file for their particular project. Those tools are being built right now, but it's work.

It's a you know, it's a bounded software development project to build that particular tool. The next thing you need to do is work with the major package managers to make sure that the Ingress and egress the software Building Material metadata happens. We inconsistency consistently.

This is not happening today. But again, it's something where it's a pretty straightforward software development project within each of those things and we just need to get it done. Finally in build systems where you have, you know all the software coming together and you can actually very on in a very automated way spit out a software build materials file.

We need to make sure that major build Systems Support software bill of material metadata and a seamless way and then for every organization in the world that has their awesome custom build infrastructure. We need to provide training to make sure that those build systems have seamless ways to support software bill of materials metadata. It's slog but it's totally doable and it's something that we need to have happen because regulators and end users are going to expect detailed software building materials metadata in these projects and services.

If we build the software that automates the distribution of this across the supply chain. It really becomes a very low cost part of how software gets built and again, And sort of commoditizes a lot of tooling out there because now you really have this metadata that allows you to get manifest that you don't have to do complex scanning for buy six tools to do and then helps that industry get to higher levels of impact in what they're doing. Then finally the last thing we need to do is just work with major open source build systems package managers to make sure that they have best practices around software distribution.

I won't go into too much detail on that. But again, we know who these package managers are. We know generally who supporting them.

They just need some additional funding and assistance to get it all done. So here's the punchline. What is all this stuff going to cost?

Here's our estimate of what it will cost for us to go out and just do the hard work both to coordinate it and conjunction with industry and to directly fund developers who need to do this work because they have, you know, very select expertise that's required to get the job done. And when you look at this in total, we're talking about roughly 150 million over a couple years to massively raise the Baseline in our Collective cybersecurity and one of the most important ecosystems, which is open source to me. That seems like a bit pretty good deal.

But of course as the person who has to go out and raise all the money and has raised millions and millions of dollars for the open source community and maintainers and projects. It's never easy. I think the A way to think about it in 2022 is to look at this number.

Just this is just one random number that I picked out in terms of the cost of a security vulnerability. Does anybody know what this number is from? This was the FTC fine levied against Equifax.

For the Equifax vulnerabilities and and security issues one organization. This was remembered Apache struts and everybody's financial data was out there for everybody. I like this terrible just one organization a 700 million dollar fine.

This doesn't count all the endless like pulling your hair out and staff hours that I am sure as I saw so many hands like dealing with log Forge like that that is in the hundreds and hundreds and potentially billions of dollars in terms of what it takes to remediate this stuff. I'm just talking the fines here. 150 mil to potentially prevent another one of these seems like a bargain and let me tell you this ain't stopping the FDC has clear guidance here that they will take enforcement action if people don't get their act together on this stuff.

And so what we're asking the industry to do and what we're working with these organizations to underwrite is join us in this Collective surge of work. To go and collectively raise our security Baseline. It for the first time in my career working in open source, which has been a long time.

There is an industry will there is a detailed consensus plan the 10 points. I just showed you if you go to the open source security Foundation website. There's a detailed white paper on this plan.

You can go in and read exactly in detail how we're doing it. But if we can get organizations like yours to join organizations like these to get in partnership with people who are generally at CEO minus one in the industry to pay attention here and to have the will to create the secure code and culture and get this work done. We can do this.

I am more bullish on this than I ever have been. But we still do have ways to go in raising the resources working with open source community members to get this work done. So I'd like to invite all of you to join us.

We are easy to find if you go to the open source security Foundation website. If you go to the Linux Foundation website, there's contact forms there. We would love to work with you, you know, it can be in any form.

It can be in financial contributions. It can be an expertise. If you have security Engineers who can help somebody these critical open source projects that need it.

We will help you get connected with those developers in order to work together to collectively improve our Baseline. And so that really is what we're working on at the Linux Foundation when it comes to cyber security. I think I've timed my talk just right here Mark.

I'm not sure if I've gone over at all, but I can take one or two questions if we've got some time. Q&A there's a microphone. Okay, so we can take a few questions, but that's what we're working on when it comes to cybersecurity at the foundation.

My name is prateek Mishra. I lead the deaf secops efforts at ADP the guys who do the paychecks. We're a customer.

Okay, I really need you to be secure here. Absolutely. And that's that's what I'm here.

So first I just want to thank you for your presentation, right because this whole problem is really Closer to a public health problem. Yeah, right covid mosquito infestation. And what's happening is that you know companies or vendors have their little private, you know special pill that you take but in fact the problem is so ubiquitous and widespread that we really need efforts like yours, and I'm so happy to see that and I want to make another comment that This is actually not just quote limited to open source.

That's right. Right because for any larger Enterprise or even a medium-sized Enterprise all of these issues are present and I'll be just honest with you. We struggle to have uniform guidelines that are well understood well accepted right?

What kind of training is needed? Yeah, right. And and you know again, I love my vendor friends.

I've worked for a vendor most of my life but but most of my professional life, but but the vendor pitches are not always helpful. What is needed is Frameworks. Yeah, and I'd like to invite other folks from Enterprise companies here happy to meet with them and and also see how we can contribute because these Frameworks and guidelines would have deep impact on Enterprise practices.

So we really need to work with you. Yeah, I think you know the the good news in respons. Out is that I do not get we used to get pushback from vendors saying like whoa, that's kind of our territory and you know, we're not so sure that should be open source.

We don't really get that anymore. I think there needs to be you know, as I said better data sharing Baseline systems and and I am convinced that you know, the some of the the security tools that are out there in retrospect maybe haven't been all that. Right?

Right, but if you raise this Baseline, it forces Innovation, and that's what I think will benefit your work. Absolutely. I just want to add my voice to that and give you a simple example right?

There is not yet an accepted standard interchange format for third party like Mercy dependency. Yeah, that just blows my mind. You know, we're going through these data formats.

No, this this is the CPE part and this yeah, it's like no, I I it's just such a routine. Voltronic good news that that is happening. So, you know, I think again getting that industry will and consensus.

This is the first time I've really seen it coming together and I think you are going to see those things happen the good news about open source, and this comes to your comment that this isn't just an open source problem. It happens in proprietary software too is at least an open source will have a totally visible and transparent set of baselines and standards and prep best practices. What I strongly suspect will happen is those will spill over to proprietary development practices as well.

I mean for years I used to talk about how much we could learn from the trustworthy Computing initiative at Microsoft the proprietary company, you know, Steve lipner and some of the people who innovated there in the open source Community like hey, they have some good ideas about secure coding. I think if we can Implement a lot of these practices in a very transparent and standardized way, you'll see a huge impact also on the propriet. software development ecosystem Thank you.

Great to see this initiative. Thank you. Is a vendor part of the supply chain that they consumes a lot of open source and distribute a lot of models that are based on open source.

We struggle with a couple of basic stuff like naming convention. Yeah, you know you you and it's partly related to that, you know, you consume something you install something and then there is a vulnerability that is reported on it. Not the name, you know basic.

Yeah naming Discovery, you know, I tried open an open ssf Discovery tool. I for several years already. I couldn't yeah.

Yeah, you told me right this is something that came out of our original research with Harvard and just trying to take inventory of critical software projects out there that the naming conventions were yeah a nightmare. It's actually if you go in and read our census reports from open ssf and Harvard. It's specifically called out in there in terms of a need to create more standardized naming conventions in this area.

And you're also right it's not working right today. It's still a work in progress. It is a goal of the organization.

It's something we've got our eye on but couldn't agree more. It's still messed up. Yeah, and another comment and it is regarding how we consume open source.

We stick to the source and never consume any binary package and no package managers. It's just we consumed software in the source build it as part. Our production environment which is isolated to block down.

Nothing comes out from the internet and then we know that we scan it. We validated it run our own unit test on it and only then we can distribute it. So you probably work at a mid to large company sites company and I mean I couldn't agree.

Don't right don't download random binaries from the internet people. Like I don't need to say this at RSA but like there's a lot of other organizations they do do that. It's very poor practice.

There is some interesting new Services coming out Google recently announced as cloud service where they'll do essentially what you just describe right where they're vetting these different components and dependencies and offering it as as a service for maybe smaller companies that aren't able to do that. But yeah don't download binaries and run them. Well, it's part of the signature, you know, I cannot trust what I download.

Can trust the source because I can scan it properly. I can validate it may surprise you how many people do to not do that? It's just a problem for all this again.

It comes back to like education both in the practitioner side. We're very focused in the application development side now, but also in the practitioner side we need that as well. I really want to thank all of you again, we're super open.

enjoying Peace needed equally if not more to cash. Like they're just is a finite set of people who can solve some of these folks. So if you have people on your team who want to be a part of something bigger than just your organization or themselves send them more away.

We'll put them to work. Thank you.