Open Source Software Security and DevSecOps – Dustin Ingram, Google,

This is Textron TV. Hey guys. Thanks for the throw.

We're here with Dustin Ingram who's a staff software engineer for Google and we're talking about all things related to open source security doesn't welcome the show. Hey, thanks, really great to be here. I would say in the last year.

There's been more focus on this issue than any time in the previous decade or two decades combined and I think the question that people are starting to have is well, we know that there's been a couple of incidents involving zero day vulnerabilities, but just how concerned should folks be about open source security right now. What is the real level of Angst I guess or and maybe are we overly concerned at this point? Like we're really concerned.

So yeah, I think for a lot of people that have worked in sort of the space for a long time. We're feeling kind of indicated like a lot of people have been talking about open source security for a while, but it seems you're right like just within the last year or two. There's just been this incredible increased Focus partially due to some high profile attacks, but I think a lot of people are sort of Getting it like understanding how much open source underpins essentially all technology and sort of like the the gaps that exist and you know that kind of obvious improvements that we could make.

Is in your perception that the bad guys are getting more in-depth and compromising these various projects because not everything is Linux right A lot of these projects are a handful of people who are doing it as much at a love and a hobby as they do it for any semblance of actual work. So are we seeing more attacks against the components that make up our software supply chain? You know, I think it depends on what you consider to be an attack, right?

This is just this really wide gradient of what you could consider an attack on some open source software, right? So for example, like malware attacks on open indexes like python package index RPM, they're way up right but there's a text are are fairly low impact. So they're like high frequency low impact they require easier to like Miss type of package name or something like that and they're like trying to steal cryptocurrency tokens or something like that.

We are maybe seeing some more high impact events too. And that would be things like the solar winds which was you know, that was an attacker trying to compromise maybe not open source, but similar attack could be performed against open source, potentially or vulnerabilities like log for J. Right and that could have been compromised as well.

Yeah, so Google is heavily invested in open source software. What exactly are you guys doing to help address this issue? And what is it you wish others were doing to help you?

Yeah, absolutely. So yeah, I'm on the Google's open source security team the call ourselves ghost. And so this is a team that was sort of borne out of the US White House is executive order, right?

So Google made some commitments to the White House that we would sort of help the broad open source ecosystem to make Changes here to make open source a little more resilient to things like log4j and other potential attacks. So our team is a extremely externally focused team. Like our mission is to improve open source that open source that everybody uses not just Google right and we do, you know, obviously have some benefit to Google because Google uses a ton of Open Source, you know, Google decided a really long time ago that like it potentially can't be more secure to Reliant open source and use the same open sources that everyone else uses but that does come with like some responsibility to act and improve and you know contribute back to those open source things.

So my team focuses sort of broadly across all security topic. And that includes supply chain security vulnerability management doing fuzzing and vulnerability Discovery. We have a whole team that works on the Linux kernel specifically and then we do some other stuff as well.

That's let's focused on technology more focused on outcomes. So we've contributed to policy in the US and elsewhere and then we do a lot of funding as well. So funding open source foundations and things like that.

Are you primarily focused on the projects you contribute to or is this a broader effort where you're helping maintainers and contributors of all kinds of Open Source projects? Yeah, there's a couple classes of projects that Google cares about so one like we publish a lot of Open Source before this. I worked for Google cloud and I helped maintain a lot of the open source libraries that we publish as Google to help our Cloud customers do stuff with cloud and you know, it's important to Google it when we publish open source packages my publish things like tensorflow stuff like that.

That those are secure and that we're doing the the steps necessary to protect our users. There's another class too of the open source that Google consumes, right? We use a lot of our own open source.

We also use a lot of what we call third party open source stuff that comes from you know, elsewhere in the ecosystem. So we are also focused on those dependencies because they're you know, it's sometimes in critical Google environments and ecosystems and things like that. So we do maybe care a little bit more about those than maybe some of the other ones but that said like a piecemeal solution here is is not going to be successful.

Right? We sort of recognize that we kind of need to make ecosystem and systemic level changes to open source, and we get the trickle down effects of that. Right?

Like if we can Elevate the entire ecosystem to adopt some new technology your best practice or something like that. We will eventually see the effects in open source software that we consume. So I guess that third category is really like just everything and that's really where we're most focused.

Speaking of systematic changes. What is it you wish that Enterprises were doing to help with this because there has been criticism that says too many people are consuming open source software and benefiting from it without necessarily giving back to the community. And what does the form of those contributions look like because it may not always be code, right?

Yeah, absolutely. Yeah, I've seen that criticism before and you know, the thing is The whole point of open sources that it is free for anyone to use and there's there is no requirement to get back. So I said I don't think any of those organizations are in the wrong in the sense that they're you know, stealing or you know taking something that they should be paying for something like that that said It is very much a community.

Right and open source, really only works. If all the members of the community are participating to the level which they could participate right. So, you know, if I'm just like if I and I say this as like someone that's on the board of a software Foundation, right?

We don't expect everyone to contribute the same amount but for organizations that have the means to make significant financial contributions to put Engineers on working on open source projects that they consume. Yeah. Those are things that I think organization should think about doing and the thing I actually really want organizations to do is Kind of take an itemization of the open source that they're using because you probably want to do that anyway to understand if you're you know, potentially vulnerable or you know, something goes bad.

What are you actually using but in that same time use that itemization to sort of think about well what is sort of the state of these open source projects do they actually need support? I think a lot of times, you know, Enterprise organizations. They just take it off the shelf and use it.

They don't really think so much about who put it on the shelf or where it came from. And so that's something you know, my team is very much trying to do something Google's been doing for a long time is just sort of like trying to Upstream support as much as we can and but for some organizations that might be as much as just like I you know when you find a bug and you think you know the solution go and post an issue tracker and share what you've learned. Potentially fix it that kind of thing.

All right, see freshness ingredients matter for software as much as they do for anything else, but I guess the question I would come to you with is do you think that maybe we'll see some level of maturation of devops workflows as we kind of think through the use of these components. We hear a lot about debt secops these days and do you think we're the end of the day this may all serve as a catalyst for kind of advancing our efforts together embrace the next generation of a more secure approach to how we build software. A hundred percent.

Yeah, I think you know a lot of yeah, so I'm one of the maintainers of python package index and I've been saying for a long time, please don't trust anything that's on the index. It's an open index anyone can publish anything to it. I think for a long time.

It was kind of okay to just trust what was there and a lot of people adopted practices that they didn't really, you know spend too much time thinking about what they were consuming whether you know, there was Integrity in those artifacts. They're consuming that kind of thing and I think like everyone got a little spooked right and and I think that's kind of a good thing that that happened because now people are sort of looking at all. What if something did go wrong here.

What if this thing that I'm using had a serious vulnerability what's my path to remediation? And those are questions that I think for a lot of organizations that hadn't really thought through and now they are thinking through it. So not only do they have the opportunity now to think about it and adopt the practices, but, you know people are coming out of the woodwork to provide Technologies to make that even easier than it was before too and it seems like at least that even a few All steps will make a huge difference.

I mean, it's not that difficult perhaps to even just check to make sure that the component you're using is the latest version of something and there isn't some known vulnerability in that and we seem to wind up shipping a lot of known vulnerabilities in code at a simple inertia, but I mean, you know, does this have to be rocket science to solve or they're just really some simple fundamental things that can be done that will make a huge difference. Yeah, I think there's I think you're right. I think there are simple fundamental things that can be done.

For the best possible use case, right the the organization that is using a lot of modern tools using modern languages is is has already adopted a lot of best practices that have come before this like, yeah, we can definitely make it easy for them to to increase their security around open source. But unfortunately, that's not the reality of what a lot of software looks like for a lot of Open Source consumers. Right A lot of them have are far behind those best practices like my team, you know, I have to remind myself a lot that I'm we're way out on the bleeding edge, right?

We're sort of thinking way into the future about what this will should look like, you know, if you're to build a new project what you should do, but for a lot of people at the deal with all this Legacy and and it makes it hard to adopt new technology new tools need practices and that kind of thing and so I think you know, there is an opportunity there to improve but it's gonna be hard for a lot of organizations. Right on the other end of the spectrum. Do you think AI will help someday?

Will I eventually get a message that says something to effective danger Will Robinson if you use that software component. Yeah, there's some interesting stuff being done here. Now that can maybe go into a little detail about not stuff that necessarily Google's doing with this things that are happening in the ecosystem.

So one thing that you could sort of think a little bit about in terms of machine learning and AI is our it's causing so Google's had a OSS fuzz which is a fuzzing service for open source for a long time. And you know, this this service just it keeps getting smarter and smarter. Right and it's sometimes challenging to find these edge cases and you know, critical software projects that have potential vulnerabilities that we don't know about and you know, the more we can do fuzzing the more efficient fuzzing can get the better those results are be and the quicker we can find those vulnerabilities and the other thing that is some what related to a machine learning is is now or detection.

So my team has some folks that work with the open software security foundation on On malware detection practices specifically and some of that is also machine learning based right? We have a data set in it. Let's say of known malware and they follow certain patterns and sometimes those signals are meaningful right?

Sometimes we can train a model to to see what looks like malware and potentially detect that but I think at the end of the day like that only gets us like 90% of the way there still needs to be some investment in you know, providing adequate, you know humans reviewing code that kind of thing and and just like General best practices around well even even if there's a problem, how do we remediate it? So what is your sense at the end of the day? A lot of folks will listen to this and you hear some folks start to say well, maybe you shouldn't use as much open source software, but ultimately is open source software more secure less secure or equally insecure as any other software component.

I think it's secure in a different way. I wouldn't want anything that's happening to open source right now any of the focus on increasing security right now to Spook people from using open source, because the reality of open sources that these projects have been a lot around for a long time and it had a lot of contributors probably more than you know, you could if you were to build something yourself, right and the expertise is pretty high as well. And you know that just the nature of Open Source means that potential attackers have had a lot of opportunities to identify problems and it's a good thing about that.

Right? I means that if you if you're shipping something closed Source, maybe the attacker hasn't seen the source code, but you also haven't had the you know support of the collective Community to determine if if folks that had good intentious if they were able to find problems in that as well. I think open sources is just incredibly powerful in the sense that we have just a giant Community far larger body and expertise overall and then your organization could provide to a certain software library or package.

So yeah, I I would say like I find open source and think that it's more secure and encourage everyone to use open source as much as possible because I think like that collaboration is actually where most of the security comes from. All right. So what's your best advice to folks?

What would you recommend that they do if they're just starting to get their arms around the whole conversation because to your point a lot of us are further down the path but looking back in hindsight. What do you know now that you kind of wish you knew a year ago two years ago. Oh, yeah.

Sure. Well, yeah, well what I know now that I didn't know two years ago. It's just some of the new technologies that Google's been working with some of our I've been collaborating with some other organizations to develop and I I'm happy to talk about this too.

I think just sort of when we think about it at a high level. I think a lot of organizations don't have a right insight into what software they're using. So that's really the first step.

It's like just figure out what you're using what version you're using where it's coming from, right? And that might sound really easy. That's a really hard step to make for a lot of organizations.

The next logical step after that is okay. You know what you're using are there known vulnerabilities in it. Right?

So one of the projects that my team works on is the open source vulnerability format and service. So this is a shared service for sort of collecting vulnerabilities for all that open source packages and making that available to various auditing tools and infrastructure, so For example, like once you have this itemization once you know what open source you're using. Using a service like like osc would allow you to say okay.

There's a known cve here. I need to upgrade this package and and take the steps necessary, but then should take the steps in automated way that you don't really have to do a lot of work to fix that vulnerability. TVs, you know, they're it's get something that's good to be aware of but that's a CDE doesn't necessarily mean that you're impacted.

It doesn't necessarily mean that it's super high severity. So usually the right answer when you find a CDS to upgrade but it's not always the end of the world. There are other things I think I'd encourage organizations to think about adopting as well.

So one is just some means to verify software Integrity, right? So for some ecosystems, this might be like a hash that you can verify that the organization that published the software artifact is actually what you're consuming and all sorts of software installers have tools for doing hash checking another Step Beyond that would be actually signing and for some projects that might mean like pgp signing and one of the projects that my team works on with a bunch of other organizations is Sig store, which is essentially a what we think is a better model for software signing and I think you know, if you have opportunity to adopt six or you should absolutely do it because it solves a lot of problems that pgp has already and it works really nicely with a lot of consumer tools like GitHub actions and Google Cloud things like that as well. So is there some central location to go find all these tools and services because it seems like we're asking people that kind of you know by happenstance go find all this stuff.

So is there some place where you know, I can go tomorrow and say all right. Here's the tools in the path to you know sanity. Yeah, absolutely.

So I'm a little biased because I'm talking about the stuff that I think and Google thinks is really good idea and there's not really a list of those tools anyway, right? There's not a list of like what Google recommends that said, we work really closely with all the numbers of the open source software security Foundation open ssf. This is a new organization as part of the Linux foundation and all of these projects have folks from that organization from various other member organizations that that work on these together and I think the openness this stuff is essentially what you should think of as Clearing House for how open source software should be done.

All these projects are open source as well. Right? These aren't commercial projects.

These are all open source projects that solve open source software security problems and you know, six stores and member project of the openness for example, so I think I would really direct people to go to the open Seth look to the open ssf to sort of be the Be the voice for what would be the best practice here what technologies are worth adopting that kind of thing? All right, awesome. Folks, you heard it here first the tools around there.

You just got to make a little left to go find them. But if we all work together we can maybe in living a more secure software World After All hey Dustin. Thanks for being on the show.

Yeah, absolutely. It's great. All right back to you guys in the studio.