David A. Wheeler, The Linux Foundation | DevOps Experience 2022

Hi there. My name is David a Wheeler. I'm the director of Open Source supply chain security at the Linux foundation and I'm here today to talk about devops security and open source software.

First although I think a lot of you already know what open source software is I want to make sure we're agreeing on some terms. So when I say the phrase open source software, I mean the usual definition of it. It's software license to users with specific freedoms to run it for any purpose to study modify and freely your distribute copies, maybe the original or modified version without constraints like royalty payments to the original author.

If you want a larger more sophisticated definition, the open source definition is a good definition put up the open source Institute. I initiative sorry. So there's a lot of opens first licenses.

There's a few that you will typically see things like the MIT license or the Apache 20 license. If you've got software, it's not open source software. What do you call it?

Well a common thing to call that kind of software is closed Source. It's kind of an obvious analogy though open source software and you'll also see proprietary software as an antonym used. Now.

If you're in the US it's important to understand that open source software is a kind of commercial software because under the US law if it's licensed to the general public and it has a non-government use it's commercial that's a particularly important if you ever have to interact with the US government. Now why I have open source software licenses. Well, there are many reasons motivations are complicated things with humans.

But an important thing to understand about open source software is that it's licenses enabled worldwide collaborative development of that software people can gather you together develop it use it worldwide identify issues collaboratively fix it and that has led to an incredible number of advantages and reasons why open source software is so widely used And it is widely used. In fact open source offer is a critical part of the modern software supply chain synopsis found that 98% of all code bases in general and of all Android AppSec specifically contained open source software. And in fact, the average amount of of code that's open source software with any code base is anywhere from 78 to 90 percent depending on the analysis and it's increasing you can see even just in a few years.

We can identify an increasing number of Open Source components being used within a larger software code base. Now I'm going to talk about security. So let me just quickly answer one question.

I often get a lot which is is open source software always more secure or proprietary software always more secure and of course that we're always as rarely true in real life that real answer is neither. If you act if you care about security, you need to evaluate that said there is a reason that open source file for has a potential security Advantage. It's it's not crazy why people think hey open source offer doesn't that have a security Advantage?

It's certainly has a potential one. That's because the fundamental secure design principles. We're identified back in the 70s by salts or Schroeder.

And one of them is that the production mechanism must not depend on attack or ignorance and open source software better fulfills this principle, so it has a potential security Advantage but That like all things potentials are not always realize you still need to look at software before you bring it in and the reality is even software. That's well run. The reality is it's written by humans humans aren't perfect.

And because humans aren't perfect humans. Sometimes make mistakes meaning that the software we make sometimes as mistakes and some of those mistakes become vulnerabilities. That's it continuous careful review is more likely to detect vulnerabilities over time that can be then fixed.

But even a well-run project can have and a vulnerability So with that his context let me make a few comments here about about my topics here. First of all, I hope you realize that already that attackers are attacking software worldwide. It's not just open source.

It's not just closed Source. It's Software so you've got to prepare. And if you're doing if you're a devops practitioner, you should be doing SecOps whether you call SecOps or not security should be integrated into your devops practices things like you should make sure you have security tools in your CI pipeline to look for problems before they get deployed.

Now some open sir software projects do apply SecOps. I run a project which specifically does apply to have sec Ops it's a little difficult to answer the question. Hey, how many open source software projects practice SecOps and part of the reason is that many open source software projects are components.

They don't directly deploy it all in the sense of deploying software to end users, you know something like hey is a number is or I isn't a number even or odd that's not something you would deploy directly to end users. It's a component that you would typically use in a larger component. Now that said many open source software projects do produce components that enable they have psychops.

Okay, lots of security guidance. Lots of tools lots of enabling technology kubernetes being one of many many. For many open source software projects build and release of their code is their version of Ops because what they put out what what they view is, their endpoint is a little different than what often people talk about for dosac Ops that said well-run project open source software projects have CI pipelines that check security before release to my mind.

That's one of the criteria that identifies a well-run open source software project. So 11. Let me talk about open source software security and the open ssf there are literally millions of Open Source software projects.

I'm not exaggerating. I did analysis years ago where we found was in the millions, and I'm sure it's only increased since then. Um, and there are many foundations which run a number of Open Source projects and you know are relevant to SecOps in various ways.

I work for the Linux Foundation many of you will probably already wear. The Linux Foundation is a foundation that creates foundations. So there's a number of Foundations relevant to dip particularly focus on SecOps related things things like the continuous delivery Foundation or the cloud native Computing foundation.

And of course, there are the foundations the Linux Foundation, you know, Apache saw from Foundation the python software foundation and so on I can't possibly talk about them all. So I'm gonna Focus today more on the open source security Foundation because it's focus is I'm going to read it right here collaboration and working both upstream and with existing communities to advance open source security brawl that's their mission. So they were created in 2020 as part of the Linux foundation and 2022.

They switched to a member funded model on 2020. There's this thing called covid that made a lot of things hard to set up and in May 2022 the open ssf release something called The Open Source open source software security mobilization plan. I'll talk about that a little later.

But basically it's you know, here's some areas that we particularly want to focus in on the future to improve security of Open Source software. So here's what the open ssf looks like. If you look at from an org chart point of view, like most Linux Foundation foundations.

There's a governing board that decides where funding goes and Technical advisor counter weight Council which oversees the major technical decisions. There's a lot of working groups and inside those working groups are a lot of projects which are efforts that work on code and cigs which are efforts that work on things primarily other than code as well some Associated projects. Nobody has this stuff is doing a lot.

It's a whole lot to take if you try to keep track of it all so let me point out just a sample of some results that I think you may find useful right now today. Secure software fundamentals is a free course openness of score cards. This automatically measures open source software sonotype actually found in a recent twenty twenty two report that it was actually really helpful and identifying projects are likely to have known vulnerabilities openness practices badge the alpha maker, which is working in proactively fine and fix vulnerabilities in some cases by working very directly with the projects and others looking at a top list looking for vulnerabilities with tools vulnerabilities disclosure guide and some concise guides.

I'm gonna point out a little more information about each of these Because I think some of these will be very very useful. First of all, of course on secure software development fundamentals. This is a free course if you develop software, you should learn how to develop secure software yet.

Most software developers have never been told if you don't know what course to take. Here's a course it's free on average taking about 16 hours to go through with three parts requires design reuse implementation and then verification some more specific topics. And this talks about the fundamentals.

It doesn't Focus just on whether you're an open source develop open source software developer. It's food teaches the fundamentals of developing secure software, whether it's open source software or not. It does discuss how to evaluate and reuse open source software because everyone does that And when you're done if you can show evidence if you can show evidence you actually learn the material you get a free certificate and there's the link there.

It strings may seem if you want to take another course. That's great. There are many other courses.

Please take something though because developing software that secure starts with developers knowing how to do it. Open ssf scorecards. This automatically scores open source software projects on a set of heuristics what they call checks each one score from 0 to 10 paying being the best what's great about scorecards is that you can evaluate arbitrary project.

They don't need to collaborate currently. There's only works on Project hosts sit on GitHub. This is not a fundamental to its concept.

It's just it's current implementation. We have taken steps to make that took a broad Mount later. Um, here's some sample checks.

It looks for things. Like hey does the project have checked in binaries or not? If it does that's a little concerning because that means they're not rebuilding them necessarily does use things like Branch protection does it have run tests have code reviews?

And so these kinds of checks basically help you estimate the risk level of that project. I have a link there for more information. The open ssf best practices is bad project basically is a list of best practices for open source projects things that we think open source projects should be doing and if an open source software project meets those criteria, you can earn a batch there's three badge levels passing silver and gold.

Um participation this is widespread. It's continuing to grow we've got over 5100 projects over. I think it's 900 passing.

It's at least 800 passing in as of this month. I think that's a typo there. But you know, hey, well, I expect that we'll get there.

And again, there's a URL you can learn more. One of the things the open ssf has done is created a concise guy for developing more secure software it when it's printed out. It's about a page maybe a little bit more.

So it's not large and points off to more information. But let me just walk through through real quick. So first of all, ensure all the privilege developers use multi-factor authentication some attackers want to take over developer accounts in order to subvert open source software if you require the use of little MFA tokens that almost completely eliminates those threats.

5% of all those kinds of attacks basically get forwarded by MFA tokens learn about secure software development or talked about that. Um use a combination of tools in your CI pipeline to detect vulnerabilities and this just screams devops, right? Okay, get those tools in there.

The tools by themselves are less effective unless the developers know how to develop secure software, but the tools plus knowledge is a powerful combination if I solver before you select it. And then the next few of listed here are really all about doing rapid updates because as I mentioned humans are imperfect we make mistakes and although we try not to make mistakes and we take steps to detect mistakes before they get released. It happens and so users of software need to be prepared to do rapid updates when a vulnerability is found in a component you need to be able to rapidly update so that the fixed version is available and that means as a developer you need to use package managers to automatically manage your dependencies Implement automated tests.

High coverage and also negative testing this means testing for things. It doesn't do I hear a lot of test-driven developer believers who only do half the test because they only test for the new functionality they added they failed a test for the things that it shouldn't do so things like hey if the software shouldn't delete a component if it's not owned by the user then check for that that has to be one of your tests. Monitor known vulnerabilities in your software is direct and indirect dependencies keep those dependencies reasonably up to date.

So if a vulnerability is found, it's not a major redesign just to use the fixed version. Before you bring an open source offer. You should evaluate it.

Here's some simple questions to ask, you know, can you avoid adding it at all? You may already have and component that provides the functionality you need. You don't need to add, you know three different ways to do the same thing.

Are you evaluating the intended version? Is it maintained and so on and the guide actually includes some information about how to estimate these answers. I think these questions are actually relevant for all software.

I'm in the issues that I'm raising are actually true for all resolve for not just for open source software, but because open source offers the vast majority of resolver. It makes sense to kind of focus in on that and how to get answers to those questions so that you can you know evaluate that opens our software before you bring it in. That's just a few of the open source software projects.

There's a lot more and I this tries to show that in fact, there's many different projects and they fit anywhere from the developer through source code reposites your bills through packaging them up all the way to distribution. Now, how are open source software projects attacked? Let me point out too because a lot of people just think about oh, you know some developer, you know, intentionally inserts code that's malicious me it that's actually a small minority by far away the most one of the most common kinds of attacks for supply chain attacks in that process between the developer and you know, the many many steps to end users always comment something all type of squatting.

This is where attackers create a whole bunch of projects with almost the right name and hoping that someone who's trying to use the software accidentally use the wrong name. How do you counter this? double check the name it turns out that's an incredibly strong countermeasure to just wait before you bring that software in is that the software you intended to bring in and often that eliminates the problem entirely but people are busy and are aware that this is an attack.

Another kind of attack is something called dependency confusion often projects, you know larger projects offers offer brings in software for multiple different repositories different Registries, and they may use a an organizations internal registry as well as external ones and they may say bring in package food from my assuming it's from their internal registry, but then someone creates a food out in a public registry and their software went built access on intent, you know uses the one that's public and some of the one that's private attackers. Unfortunately, I've realized this and they Jake tricks like I'll create a whole bunch of packages with names that look like private packages. I'll give them version 999 and the package manager will say wow 99.

0. I've been using Um organizations thought creating internal Registries would automatically Aid security and the problem here is if they use wrongly this can make it worse. So that doesn't mean that internal Registries are wrong, but it does mean that you have to use them wisely if you are intending to bring software in from a private registry make sure that's actually what's going to happen.

One of the more recent events that's happened that were pretty excited about something called Sig store six stores been in development for a while here, but the problem is trying to address is that there's been many previous tools to cryptographically sign open source software projects, but the packages but they're often unused they're very challenging to use correctly. It's often in Practical and so six store was created as a free to use non-profit software signing service to make it much easier to sign and probably more importantly to verify because if you don't verify signatures, there's no point in signing things. Um, it uses open ID connect and that enables the use of all sorts of existing security controls such as two-factor authentication and so on.

What's interesting about six stores that enables the use of something called ephemeral or short-lived key pairs? Those who are aware of the details the technology about evolving public and private Keys. It's often a challenge to carefully secure the the secret key of the private key.

And so this takes a different approach. So that users typically don't need to manage the Seeker key, which illuminates a lot of problems and likes a lot easier to do. Um, one thing in six door enables for example is enables using Cloud ci/cd providers without them even needing to have access to a private key that makes a lot of things simpler.

It also supports transparency logs. So you can anybody can monitor the transparency logs to find out what's been signed by whom? And six stores General availability was on October 25th.

So this is relatively recent news. 5% uptime and we've got Round the Clock pagers, you know, the goal is to make it so that you know, we've been it's been in development for now for a while, but we want people to be able to use it as a service and be and be willing to trust it because it's just Worthy. I mentioned earlier the open ssf's mobilization plan.

This is something that was released in May 2022. And basically we had a number of people sit down ask. Hey.

Open source software is so widely used. Yes. There's a lot that's really secure.

But what can we do to make things even better? And so we we don't know we had started with three goals and identified 10 streams and little symbols Twisted here listed here anywhere from improving security education and digital signatures to improving the use of memory say of memory safety doing code on it's improving software Supply chains and so on so you can see the plan for more about at least where we're trying to head and the year ahead. Now I've talked a lot about open ssf because they're focused on security and that's what I'm trying to talk about here, but they're not the only organization involved in either security or certainly not an open source software or in devops.

I do want to give at least a quick shout out to the continuous delivery Foundation. They have their own software supply chain group. They've got in your probability group some best practices and what they call the continuous delivery foundations reference architecture link there.

So please go to check them out. and finally I encourage you to get involved really for any foundation certainly for the open source security Foundation, you know, please they're open ssf for example is doing a number of things that are in the early earlier stages SLSA is getting ready for a 10 release. And there's some s-bomb everywhere tool work.

There's some education work. There's work on what we're calling a metrics dashboard. We already have some prototypes.

We really want to turn this into something that everyone can use to get more information about open source our projects. We're working. We've already identified and refining identifying critical projects.

There's something called the secure software secure supply chain consumption framework. Which is early stages getting going so please get involved. That is the best way to influence these to make sure that they'll work for you that they'll be helpful for you that you know, and there's various ways to get involved an easy ways.

Just show up on a video chat for one of the alternate weekly meetings. We also have mail lists. We have slack channels.

You can also see our blog to see kind of what's going on. As I mentioned there's many many other open source software projects. There's many other foundations.

I encourage you get involved find out what they're up to more broadly. I do believe that industry Academia and government really should work together. It's important because open source software is just widely used.

It's all over trying to say. Hey, we're going to get rid of it. It's not really a sensible approach.

It's a lot like we're going to stop using computers. We aren't Um, and a lot of Open Source software is in really good state state some is less. So and so we want to work to identify areas where either things are not so good or just we would like things better.

And again the best way to influence an open source software project direction is to get involved and with that. Thank you very much for your time and attention. I'm happy to answer any further questions that you have and if outside of this this Gathering you want to ask or talk to me further.

org. So, thank you very much.