Yesenia Yser and Jonathan Leitschuh, OpenSSF | OSS North America 2023

This is Techstrong tv And welcome back to the Open Source Summit here in Vancouver. We're here with Insignia and Jonathan and somebody named Sassy. And these folks are from the Open Source Security Foundation.

And we're gonna be talking about some of the research they've been doing and some of the work that's going on in the project. But first, I kind of wanna address the duck in the room. Jonathan, what's the story with the duck?

So the duck is, uh, was originally a gift from, um, BSides Las Vegas. Uh, it was a request when I was speaking at BSides last year when they had the CFP of, you know, what unusual requests do you have of the main of the organizers? And, um, from a previous conference, I'd been given a duck and I said, I'd love a giant inflated rubber duck up on stage while I'm presenting.

And I didn't tell my co-presenter to this and I'd forgotten this. And the middle of BSides Las Vegas, while I'm presenting on stage, one of the organizers walks up with a box, opens up this duck, and begins to inflate this duck in the middle of my presentation while I'm presenting live on stage, man begins to lose. It's, it's a large duck.

He ends up starting to turn purple. He ends up sitting on the corner of my stage while presenting, while I'm still presenting. And, uh, my co-presenter has no id, he's like totally confused.

Finally, he finishes and walks up and says, here, here is the duck you asked for. So I brought Sassy with me on tour, um, speaking at conferences from, um, all over va, you know, three conferences in Vegas, uh, to, um, you know, Stockholm to Italy, to Japan, to, you know, to San Francisco, to now here at, at the Open source summit. So I've been giving the same talk all over the, all over the world and, uh, and Sassy's been the companion that I've had for this journey of telling this story.

So, yeah, Be careful what your asper Yeah. Okay. Borrow the story.

Yeah. And Sydnee, you guys have been looking at some of the open source projects out there and the security and the practices. What have you seen so far and what, if anything, surprised you?

So, definitely seen a growth of, uh, interest and movement within security on our open source. So different working groups within the open ssf, uh, tackling different areas of cybersecurity. And then he said, what scares me Or surprises either or surprises Me, um, the amount of tech debt that we still have in the industry, in the world across open stores, you know, in corporate organizations, just the tech, the amount of technical debt.

And it's not just insecurity, it's quality assurance, it's documentation, it's just representation of the software as it is. Nice. And you've been doing the research, so what did you kind of find and what are you looking into right now?

So The work that I've been doing, um, uh, started somewhat last year. Um, it started before this, but, um, I was, uh, awarded last year, the first ever Dan Kaminsky Fellowship. Um, Dan Kaminsky was a famous security researcher, um, who tragically passed away in 2021.

Um, he's very young. He was 42. He was best known for vulnerability in DNS back in 2008.

Um, but he was also known as an individual that was very passionate about security, very passionate about securing open source or securing, you know, the industry for the people that didn't know about technology. Right. Um, and he had a, you know, heart of gold.

He was, he was an incredibly kind person. And so when he passed his company created what was called the Dan Kaminsky Fellowship to commemorate his memory. And, uh, I submitted a proposal for this idea where I've been bulk generating security fixes to fix vulnerabilities at scale across open source.

And, uh, I was accepted to become the first ever Dan Kaminsky fellow. So I spent the last year previ, you know, all of all of, uh, 2022, um, engaged in working on automating, fixing common security vulnerabilities across open source, uh, using a technology called Open Rewrite. And so Open Rewrite is a format preserving abstract syntax, trade transformer.

And it allows me to, uh, target, you know, specific vulnerabilities. This is not like depend bot, this is more like actual vulnerabilities that appear in the source code because someone post a question on stack or flow and the answer was vulnerable. And so everybody's copied and pasted that same answer across the like, source code.

So how can we go out and fix these same vulnerabilities that appear again and again and again and again across multiple projects at scale? And so the work that I've been doing is, you know, for example, I generated 165 Polar Quest to fix zip slip, which is a very critical security vulnerability. Um, unpacking zip files, um, across the Java ecosystem.

Um, and so engaging in work like that of like, you know, greatness vulnerability is one place, but it's probably a lot of places. How can we just go eliminate it from the open source ecosystem? And, and the work now with the Alpha Omega project is we, I all those campaigns that I've run have been one-offs, right?

It's been, I've run this campaign at one time, one snapshot in time, um, but people write new code, new code gets introduced. So the goal with Project Alpha Omega is to not just have these vulnerabilities get handled in, in one snapshot of time, but also like, let's go eliminate them. But let's have these campaigns running continuously monitoring open source software to detect these common vulnerability patterns and just go squelch them immediately so that they, they stop appearing in our software industry across the entire Ford.

Going back to that tech debt. Yeah. Tech debt.

Yeah. We talked about Alpha Omega yesterday, but not everybody watching knows what Alpha Omega is. So take a crack in what it is exactly.

Alpha Omega trying to do my, My, one of my favorite sales pitch nowadays. Uh, so Alpha Omega's actually the beginning to the end. So, uh, it's split into two different portions.

You have the alpha side and the Omega side. Uh, the alpha side is more focused on, uh, engaging with foundations and open source organizations that are already running and working with them to either grant some funds consulting and providing the means to be able to be more sustainable in the future. So let's say Alpha Omega runs out of funds, they're still able to sustain the security work that's needed within, uh, their organizations to continue, uh, improving the ecosystem of the open source.

Um, and then on the Omega side, essentially the TLDR is automating and scaling Jonathan. So essentially being able to automate vulnerability identification, triaging engagement with our maintainers on to the remediation of the security vulnerabilities across targeting 10,000 critical open source projects. So those open source projects are, uh, a mixture of, uh, the information from one of the working groups.

So the working, the, what is it, critical software? Yeah, The critical software, uh, sorry, The list of, uh, hundred hundred, uh, uh, projects. We took that list.

We've also used a wide range of different sources, like one of the hardware ones and communications with different organizations to provide a list of 10,000 critical open source projects that we're targeting. We are open to, um, adapting and modifying that list because, you know, everything in the software industry is very iterative. Um, and then I don't know if Jonathan wants to add any more to that.

Yeah. Just, you know, for Alpha projects, we've, we've been fund, we've funded like security audits of, you know, jQuery, which is a critical piece of software that's absolutely everywhere, right? Right.

Um, uh, they've been fund, you know, alpha Omega has been the funding source, uh, source for, um, uh, I think they, that, um, the, uh, no JS project now has a full-time software, uh, security person on, you know, that is, is paid to, to engage and, and focus on security for no js. Like that was not possible prior to this. So the alpha side is, is is focusing on like we've, we've identified these projects.

They are critical, they need more resources than can be, you know, than one researcher can provide. Like we need to throw a full-time staffer probably at that project. So a team, we've seen Teams being grown out in some of the organizations as well.

Yeah. Is it your sense, and neither one of you can answer this, but the vulnerabilities seem to be fairly common and a lot of 'em are low level and the same mistakes are made over and over again cuz people are human. Are the bad guys aware of all these vulnerabilities?

Are they targeting them? Are they looking for 'em? Or, you know, what is the level of awareness and, and, and, and the savviness of the opposition as they say, they're Definitely aware and then they're getting creative with it.

Like, um, there's different levels of, you know, attack vector. There's different areas that they can actually focus. Um, and leveraging maybe one or two common vulnerabilities actually gets them into a higher area of software.

So, um, There's This common ad is just like, it's not Vulnerabilities make, just are just vulnerabilities. They're ways of, there's ways of exploding systems, but attackers are using chains of those vulnerabilities, right? So it's, it's, you know, they're leveraging the first one to get in to get access to being able to use the rest of them, right?

So if we can cut it off at the beginning or Well, if we can, if we can clean up the, like all of them, right? If we can clean up the low hanging fruit, it raises the bar for the entire industry, it raises the security level to make it difficult, more difficult for, for the rest of, you know, the next attacker. Right.

Um, do we have direct statistics about, uh, you know, open source being directly targeted? Well, log for J showed that, you know, any, any, any widely enough use piece of software is, is a, is a great candidate for, for a widespread exploitation. Um, is that any worse than corporate software?

Well, no, we've, we saw, you know, um, uh, for example that, um, exchange, right? Microsoft Exchange has been widely exploited recently. Um, you know, and it, it, it's just, it's not, it hasn't, not a lot to do with about just open source or not open source.

It has very, very much to do with like how widely that piece of software gets deployed across the entire industry, making it, making it a rich target. Mm-hmm. We talked about the mistakes that developers make.

Truth of the matter is cybersecurity was an elective when they were being trained and most of 'em did take it still is, or not even available, right? Yeah. So Are there particular courses or actions you think developers you would recommend to them to say, Hey, if you want up your security game, go look at this or do this.

I usually recommend for my mentees, uh, the o os top 10, and then to jump on to a flirt sweep. And then ports port weer, I believe has an, uh, has an actual hands-on tutorial that you can get your hands on and actually either perform the vulnerability and identify like how it works and then also patch it. And they have a widespread of learning paths in different areas that tackle not only just the top 10, but different areas of security.

So it's usually my, my go-to for folks that are interested in security and either from a software developer standpoint or from, you know, just curiosity of like, Hey, what is security? What are these vulnerabilities I keep hearing in the news. I'm like, let's start with the very basics.

Just go check out your top owas top 10 for either mobile application or web application. And then check out, um, be sweeps, port swagger. I am not a sponsor of them, I just, I point to them because of the, the rich information that they have in their tutorials.

I think A lot of their, I think a lot of their labs are free too. Like they're available. Yeah.

So you can just, you know, 5% you can, you can take, you can download the, uh, the Burp Suite community edition, which is also free and use it to, to prove, you know, demonstrate like, and, and get hands on experience exploiting those systems. You know, that's very much, you know, live deployed systems. Um, for, for software vulnerabilities, like if you wanna look at, if you wanna look at, um, getting better at like auditing in source code, um, I recommend, you know, tinkering around with code QL is one of the technology you can tin around with.

But then also O OSP has a project called, um, web Goat, which is a, a a, an open source project, which is a known vulnerable project. So it has a bunch of vulnerabilities in it that you can, you know, download it and play with it. And then also if you wanna inspect the source code to try to find those vulnerabilities as well, you can do that there too.

My favorite website growing up was hack this site. It's been up and down over the last couple years. And then Hack the box is another one that I've seen.

Yeah. Very popular, Which may be used for good ill, depending on what you do with it, right? Right.

But that's all, all this knowledge, all this knowledge is, you know, it's, it's, yeah. Even Alpha Omega, that's one of our, you know, our risk factors is we're doing it for the good, but the bad could use it as well. Yeah.

People are sometimes, shall we say, inherently lazy. So won't they just wait and for you to come and identify all these things and then automate the fixes for 'em and, you know, and then send you a nice note and saying thank you if We get that thank you note. Um, so, um, maybe, I don't know, like, it, it, I think that, that's an interesting question.

I've never, I've never had that one before. Um, uh, I think that, uh, in general, like even with a polar request that we're generating, we're adding work, right? If they can do it right the first time, they'd rather do it right the first time, you know?

Right. And then also developers are hopefully gonna be using our, we've seen more and more commonly used like, um, copilot get up, copilot, things like that for development, right? If, if those tools that are already helping developers write code faster and more productively can also start generating the code that's secure when it's, you know, giving you the suggestion of like the next 30 lines, right?

Mm-hmm. That's, that's the, that's the right, let's just get it secure in the beginning, right? Because, and, and I also, I think that there's, there, there's a, there's a decent understanding in the industry that, like a vulnerability fixed earlier or a bug fixed earlier in the pipeline.

This like shift left thing. Like if we do that, we end up, like it costs a lot less. It's a both from like the customer's perspective, from the overhead perspective, from the, so I don't think that a developer is gonna be nefarious in the sense of like, I'm gonna like intentionally introduce a vulnerability cause I don't, or like, even like, not ne but like intentionally not learn about this stuff.

Reckless. Reckless, right? But I think that it's just like, it's, it's not about like, I know that somebody's gonna come and fix this.

It's more like it, it more comes from the a point of view of like, they don't know it from the beginning. The beginning. So I think that's, that's a, the bigger part there is like trying to get security into our, into our education, trying to get concepts like, um, testing, testing you.

I like, we've been, we've been interviewing a bunch of people for jobs and I've interviewed and, and one of the things you ask is like, what is your exposure to unit testing or integration testing? And you got a lot of people coming outta college with the answer of, we've never written any tests Yep. In our entire, in our entire college.

And I'm like, I did. Yeah. Yeah.

You know, so, so making sure that that, that those basics are definitely part of the fundamental careers Yes. And fundamental education process that, that they're coming outta college with. Yeah.

I wouldn't necessarily say they're lazy. I think it's just restructuring our education program to essentially cover the different areas of the software lifecycle to include security, include testing, including these very important features that are involved in the software development lifecycle. Like you have site reliability engineers, um, I'm in the process of deploying a software and I'm like, not sure what's going on, but I can develop features, but I'm struggling with the, you know, the operation deployment.

So going back into our education and restructuring it where they get a full knowledge and a full structure of how to do it. We'll reduce this quote unquote laziness. And then it also goes into the industry, right?

Industries are more focused on revenue and pushing out the next product for our customers Without getting a pen test done first. Right. That they're gonna be skipping these steps so that they can make their deadlines.

And essentially it's not being lazy. It's like, Hey, um, big boss guy, you know, this big boss person folk at the top just needs this, you know, we got so and so and so and so customer, let's just get this out and we'll, we'll throw that into a tech deck. Yeah, yeah, Yeah.

Black Hole bucket. I have, I have run into places in my career where I've been like, this really should get a pen test before it ships. And they're like, we need to ship it.

And I'm like, but so, because the industry, every single time you agree to, to buy a piece of software, right? You're usually sign signing an agreement that says the software is delivered as is. There are no guarantees, yada, yada, yada.

Right? Because the indu, there's no regulation in place to, you know, say, well what about secur? You have to have, have a base level of security that, you know, it's just allowed to ship without there being a risk that there could be repercussions for shipping that software without having done those pen tests with before having done those audits.

And that's on the corporate side, like that's a, that's capitalism failing us. Mm-hmm. But we have historically tested everything from cars to Yes.

Um, bridges Everything before they went in airplanes. So how did we get to the point where we just kind of stopped thinking about testing for software? I think that goes Back to the revenue, Well it's regulations too, right?

Like government has been involved in regulating these industries and there's no regulation in most of these industries around software shipping. So I know that there's a lot of negativity around the regulations around the EU and EU attempt to regulate open source. So I'm not for what that is currently stating, but I do think that there needs, I've heard it said there's two different ways that I've said that this, this, this shift could occur.

It's either government regulation or insurance companies saying, we're not gonna ensure this software unless it's had a te unless it's had a test, unless it's been audited. You know, that sort of stuff. So there's, there's two potential angles where, you know, capitalism can come in and say, you know, this needs to change otherwise, you know, you're either gonna have risk from government regulations or insurance is not gonna ensure this stuff.

Or you as a company, unless you've had, had these things, have had these basic level of security performed For testing. Yeah. This is an interesting area cuz I was actually having this conversation a little bit earlier, is, um, some of the times it becomes a check the box security.

Yes. Right? It's, we need to get this out.

Um, we need to make our auditors happy. Let's just like say we did and just throw it out there without the proper testing, without, you know, essentially, essentially when the auditors come is when everybody scrambles to make it work and actually follow what the process is from, uh, several of my experiences where, but, or you know, Just people, you know, people I know people I know nothing with my past experience, but it, it's just that, it's that, that check the box. Let's just make sure that we say that it's, it's there, it's available.

Um, and we met the bare minimum. Mm-hmm. I've heard a great, is there some way to think this through where we're automating the test more beyond just the check box so that we are actually executing these tests within a, some sort of DevOps workflow for these organizations?

And is that the level of maturity we need to get to? Yes, I think it's a good baseline, right? Like, you know, having automated testing is a good solid baseline, right?

I do think that there's really a human element involved in any, in any testing ESP because they're, we have yet to reach the point where, you know, you can encode a hacker into, into a box, right? Like, you, you really need Lot money. They're trying to do that.

Now. I know I, you know, I That's the ethical Hacker. I know.

So like here, from my perspective, right? The reason that I'm going down this road is like we have so many of these SaaS tools, right? The static code analysis tools and they all scan for these vulnerabilities, right?

And they, they're, they're getting the AST nodes in their hands to give you the alert and then all they do is let them go with the announcement of, you probably have a vulnerability here. And my annoyance is, you've, you've done the work to get the AST nodes held in memory. We know that these are the places where the vulnerability exists.

Just rearrange them and fix it. Mm-hmm. Like, you're, you're so close.

And so that's my goal with Open Rewrite is to just make that extra step that like, let's just finish that story to, and we fixed it here. So one of the goals with the analyzers to become, uh, a piece that they can throw into their ci cd as well because it has 20 plus different security tools. Some of them are licensed.

So you may not be able to get those, um, unless of course you get the license. But going on to what Jonathan says is sometimes you get the report and the report is like a hundred something pages. I don't know about you, but I don't need a, a reading material for bedtime.

Yeah. So you do Containers can't handle that either. You, you Can't handle that load, you can't handle the, uh, each of them is different and each of them requires a different level of knowledge and complexity and understanding to sit back and it's like, Hey, do I fix these 10, uh, critical and 80 important, or do we ship this product and make that $2 billion sale?

Mm-hmm. Um, capitalism is gonna go, It's not relevant, not as relevant in open source, but definitely relevant in commercial space. But Do developers tell themselves that, well, we're using all these agile methodologies, so we'll fix that in the next update that's gonna show up in the next two months.

Yes. We'll put it in our security sprint. This is the one I've heard.

Have you really? I've heard Security sprint. Oh lord, That must be like a DevSecOps thing now, right?

I had that same thing but a while ago with respect to accessibility, I was like, you know, reading, reviewing a poll request be like this, this is definitely not screen reader supported. Cuz I, I also spent a bunch of time doing screen reader accessibility support and I was like, yeah, yeah, that's an add. And I'm like, no, no, no, no, no.

You get the accessibility in while you're writing it the first time and you have to tuck the code again. You know, do it, do it right the first time. All Right.

You've been given the magic wand and you can fix anything. What's the first thing you're gonna fix? Ooh.

Yeah. Work yourself outta the job right there with that magic wand. Do you wanna go first?

Cause I'm still processing that one. Um, now I know why my group was silent on my paddle the other day. That's, that's a really interesting question.

Um, how powerful is this magic wand and what's the scope of it? As Powerful as you want it to be? I don't Know.

It's fixed world hunger. Well, there you go. I was hoping for something.

I know, I know, I know. I, you know, um, I would say education. I think if we have more folks educated, more folks in the industry, we have such a gap in just talent and diversity across, we have so many open positions with not the enough folks filling those gaps and being able to actually participate in and grow their career in that space.

That I think, uh, if we can fix one thing would be the education system to get more folks in the building on the software, developing it with the proper well-balanced knowledge of how to produce software. Because there's a difference between a software developer versus a software developer that security conscious versus a security, a software developer that's security test conscious. Mm-hmm.

Agree. There's no need to hack software anymore. I dunno, I love the exploitation, right?

Like, you know, take away, take away the incentives, but like the incentives are always gonna be there, right. Got companies, corporations, you know, is, um, There's always gonna be a way, there's the will will always find a way. Yeah.

Yeah, yeah. Unfortunately. And that's how we keep our jobs.

I know. It's, it's human, it's human nature. So I Don's just your job is a little too easy right now as part of the, I don't know if it's too easy.

Yeah, yeah. It's a far it's a heavy load. Yeah.

Folks, you're hurting here. As always. When we have a problem, it has something to do with our education system.

So maybe we gotta take a look at everything from the very beginning and start over. Folks, thanks for being on the show. Thank you.

Thank you. All Right. And we'll be back in a minute.