Software Supply Chain: It’s All About the Code | Predict 2023

Hi everybody. This is Mike Rothman in general manager of tech strong research here with our next panel in the cybersecurity track software supply chain. It's all about the code and I'm like really fired up now because I've got a great panel of folks here too illuminate your idea as we were talking you did the trends earlier obviously during our conference keynote.

The thing that capped really coming around as one of our key research focuses and one of the key things that you have to worry about all year. Is this idea of software security. How do we make sure it's secure?

How do we make sure that our developers understand what they should be doing? How are we going to secure these pipelines that everything runs through? What are we going to do about software Bill materials and a lot of the emerging regulation that's there.

What if I were applications don't even work from a security standpoint and we have to supplement that with other techn. Allergies, we're gonna hit all of those topics and more but first let me bring my panelists in so welcome everybody. I'm really happy for you to be here.

Let's kind of just go around the horn a little bit introduce yourselves your company, you know some, you know not I don't want to walk on the beach and I really Um, you know, really just kind of a little bit about you for a couple of seconds and then we will jump into our first topic Tanya. Why don't you start off? Hi, I'm Tanya Jenka and I'm the CEO and founder of wehack purple an online community podcast and Academy where oh we talk about is securing software.

So this is the right place for me. Thanks and you were picked specifically for that. So Tanya and I know each other for a little while and I was putting the panel together was like there's nobody better to really kind of talk about the developer side of the equation.

So Tanya is really going to help us out on that front David. Why don't you go next? Thanks, Mike David DeSanto.

I'm Chief product officer for gitlab prior to going into product. I was a full-time security researcher and developer. Very passionate about this area.

But yeah, it's employed my creative side to go to the dark side and moved into product. But I think it's allowed me to have a unique perspective on what developers are struggling with as a former developer myself. Yeah, especially on on the security side, right?

Because you know bringing that, you know kind of hackers ethos into you know, again kind of building an environment to you know, both be the repository as well as kind of ship code all the way through to deployment. I think that's an interesting and unique perspective. Next is steam panelists, Mr.

John Pescatore, who honestly I'm not even gonna kind of venture to guess how long I've known John but let's just say it's measured in decades as opposed to years at this point John. Welcome introduce yourself my friend. Thanks.

Welcome everyone. Yeah. It's been a long time.

I'm John pescatory that's 10 years. I've had a fancy title at Sands is the largest infosec training company out there. If you don't familiar with us and I spent almost 14 years before that is the lead security analystic Gardener and started my career out at NSA and the Secret Service and sort of the government side of all this and just one real quick an adult when I got out of college and went to work in NSA that an internal newsletter called crypto bites that's declassified now.

And in June of I'll say the year 1978 I've read about buffer overflows in the nvs operating system and how they were exploitable. So we've been working at this for 50 years. I think we can make some progress now, we've made some progress but you know, I think it's safe to say it is Groundhog Day to it to to a certain degree where we keep dealing with the same things over and over again and our last panelist Jeff Williams from contrast security again, you know kind of been one of the early, you know, kind of folks a big leader in owoss when that first got you Jeff.

Let me think. I remember that did you use yourself a little bit too? Yeah, Jeff Williams.

I'm a CTO and co-founder at contrast security. We sell an application security platform. It's absolutely Groundhog Day.

Actually, you know, we have made some progress, but the problem is that the situation has gotten much more complicated faster than we've made progress. So I actually think we've gotten farther behind since we started OS back in 2002. You know what to do.

We have a lot of work to do for sure and we only have a fixed amount of time that we have, you know with this panel. So let's kind of Jump Right In. So what one of the things we did this year at predict is we introduced a set of Trends and each one of our coverage areas security is no different.

We have five Trends in security the first one being software security and and the whole concept was really this idea of shift right question mark right, you know are we at the point where developers are starting to push back on, you know, the fact that a lot of folks have just dumped all sorts of responsibility on the developers security being one of those things, you know, are they starting to push back but really some other key questions start to come in there too, which is what degree should we be training developers in order to you know, kind of start thinking about secure code. How do we ensure that when we have some issue in the pipeline David? I know you'll know quite a bit about that.

How do we create some urgency to make sure that those defects get As opposed to just you know kind of buried in the in the inevitable scrum in Spring as that gets planned. So we have a lot of issues to really deal with around kind of how do we interface with developers more effectively to again push this agenda of secure software. So Satani I asked you to take the lead on this discussion since you're dealing with developers pretty much all day every day.

Yeah, I think a lot of developers really do care about security but they also care about speed and accessibility and if their client is pleased and if they made their deadline and we're just one of many important priorities, I saw a talk by John where he said don't shift left shift everywhere. I was like, yes, and then I remember Jeff and I having a conversation and him saying that too because security folks start saying shift left which means start security earlier in the system development life cycle, but it did not mean don't do security the rest of the time it did not mean We don't need to monitor. It's out and proddle just leave it forever.

And I think the message got a big contorted by perhaps some marketing professionals. But anyway, I digress I think Deb's really do care about security. I wish that with every boot camp and every computer science class and every like Learn Python in 30 days, whatever that they started with security being a part of each thing.

So we're gonna teach you to get user input cool. Now you can get it now. Let's talk about risk.

Maybe that input isn't so friendly. I feel like if we built it in from the beginning we could get a lot better results, but we're literally graduating millions of different people into it every single day and all those universities and colleges are not that interested in security, unfortunately, so I think that's one place where we could definitely definitely improve. What about everyone else was everyone else think Doves Care So I believe they do care I think in my experience it's about giving developers the tools.

They need to have to be successful. I I like to say like developers don't wake up in the morning and say today. I'm going to write a zero day vulnerability in my app and then go yes at the end of their day right like that.

That's not how they operate. And so if you can make security approachable, then it's less finger pointing where the security team says. Hey, you create all the vulnerabilities developer like be better as other developer saying like you gave me this vulnerability a month after I did my work like I don't remember what I was doing that day, right?

So I think it's really if you give them the tools they do care about security because dealers want to write the best code and that best code include secure code. Yeah, I think we have to give it go ahead John no good. I should say I think we've got to get developers better tools.

I don't think we can train our way out of this. I spent many years teaching secure coding classes thousands of developers and Tonya, right? They do want to do security right for sure.

They're interested Securities cool, but It's not really feasible to teach them about all of the Thousand cwes that are out there that they're they're all complicated and tricky and modern Frameworks are really complicated. Like it's hard to know which pieces of data are input and which aren't and which things are dangerous like I guarantee in 15 years of teaching developers. I never taught anybody about log for shell.

Because it was very hard to predict that sending a untrusted piece of data into a log file would cause a remote code execution. So we can't do it that we've got to have better tools to allow developers to write code without making it so dangerous for them to do it. Yeah, I think it's also we spent a lot of time talking about training and developers better tools good stuff.

But you know think back to when Microsoft finally got security religion right around 2002 when Bill Gates wrote his famous memo. I was spending a lot of time at gardeners sort of tracking them and it wasn't that they immediately had a secure software development lifecycle. But what they immediately had was the product managers started getting judged your product may have shipped but it didn't go to market if it had these levels of bugs when coming back that required really really expensive patching and really really other expensive things from Microsoft could do to test that they would work.

The cool thing is seeing across the Sands Community is I call it the younger cisos, but more and more the software Architects are coming to the ciso and saying hey we got this thing called privacy. It's one of our requirements because we're competing in a marketplace where we got to prove it or Europe and gdpr and blah blah. You guys want to join us on this versus the security group having the force their way in so things like privacy instead of calling its security instead of talking about bad guys talking about meeting people's needs to protect Grandma's stuff is a way to get the requirements in and get them doing built as features versus annoying things the annoying security people want.

Well, let's dig into A little bit right because I think the highest profile company that has really tried to use security as a major differentiator is of course our friends at Apple, right, you know kind of that's been one of their Hallmarks of you know, kind of well Google is bad because you know, the App Store and all that other stuff is total crap and you know, you'll just get compromised. If you use apps out of that environment, whereas hours is pristine and ours is safe right now. I would guess that a lot of folks like they're blue Bubbles as opposed to a green bubble.

So they buy an iPhone. I mean, do you know a folks that have actually decided to buy Apple products because they perceive them to be more secure. I didn't yeah last year I did but then I got an Android phone because I start having it was just making me so crazy.

I wouldn't let me do any of the things I wanted to do because all people on this call are super users and we have used our devices but I definitely switched because of that and then couldn't stand it. I am not the normal person. Um and me buying things because I want my personal life to be more secure.

I don't know if that's a standard we can across like I think think of the Met think of the giant leap. We just made here we went from talking about, you know, I was talking about Windows. We're impossible to have a white look we can never make white listing words.

Now, we're talking about the choice between two app stores to White lists. That's right. And I asked for directors all the time.

Does it bother you that? You can't run every app you want they're like, no. Why would I want to We've that's a hugely leap up.

We've made on this Pretend This is an iPhone that we've made on these devices that we fought for years. We could never get done on the user desktop people at home are working on are much more secure than they are at work on the PCS. They're using at work and apps apps that goes.

Hey John, you got a Microsoft back in the in the 2000s and one of the things that they did that was I think really amazing was introduced aslr and debt. So they made the platform resilient against developer mistakes and Apple's done the same thing to a certain extent with things like privacy and permissions and so on and I think we really need to do that. Same thing with other kinds of applications like web apps Web apis currently those environments have you know, all the developers code pretty much runs as root with really powerful capabilities and we can make those things a lot safer by putting it in the platform.

It's the only way to scale Software Security Solutions make things better at scale. Yeah, that's an interesting point Jeff because you know, and again a lot of people will sit there and acknowledge the fact that we build a lot less of our applications and we compose and assemble a lot more of our applications nowadays using all sorts of different libraries and components and third party services that are connecting into our application environment, right? How do we start dealing with you know, kind of it's not that we're just responsible for our own developers, which are hard.

It is hard enough. What about everybody else's about because we're Using their components, right is it? Is it just handing all the stuff when it comes in?

Or you know, how do we start to make the environment more resilient knowing that we're not actually in direct control over a lot of the code that ends up, you know being shipped as part of our applic. Yeah, I think there's a couple parts of that. I think the first is really understanding the components you're pulling in.

I the one thing I always am surprised by if you're talking since we're talking about developers, they want a certain package included so they can get this one functionality and they missed that might have pulled in like a hundred another packages and so giving developers security teams operations teams better visibility into what's actually being built as part of deploying their application that begins to help with that. I think the other component of that is a lot of what you're seeing with software supply chain security and being able to have you know, sign packages that you know that they're original. They're not one that's been tampered with as well as being able to sign your own packages.

They think about the salsa framework and being able to say like no, this is really the thing that I say it is. I think those go a long way to it. I think the other really big component of this is security is not one person's job at the company a couple people in the panel mention that I don't think you can stress that enough.

You've got developers who feel this honest to write better code. You've got security teams who are up at night worrying about the risk, you've got operations teams or sometimes stressually thin but everyone together if they're all focused on working together to secure the application the environment companies are more successful than if it's a whole person's job. Yeah, I agree with that.

And any other comments is we're kind of wrap up here points. Yeah. Hey Mike.

I just want to mention so that you started by saying an applications are or composed more than written, and it's not really true. If you look at a modern application, it's two-thirds custom code and one third. Software components that you get from somewhere.

That's if you look at the code that actually runs now there's a ton of other dead dependencies that get Dragged In but if you focus on the code that's actually runs and actually risky two thirds is custom code that people have written and so a lot of the risk comes from there. You think is you know, like generating an s bomb and like just knowing what components you use isn't really going to tell you very much about whether that piece of suffer that application or API is anywhere near secure because I mean it's just like looking at the list of parts. You could build something really awesome out of those parts, or you could build like a bomb out of those parts.

It's hard to tell just by looking at the list of ingredients, right? Yeah, and that's right. And actually that's a great segue into kind of the discussion.

We really need to have about you know, kind of the underlying infrastructure for how we process code. Right as it goes from something that comes out of a developers IDE into the repository into the testing phase into you know, kind of the deployment and the operations piece of it right David you spend a little bit of time right dealing with customers is there, you know kind of dealing with with their code and moving it, you know through these pipelines. I love to get your perspective on what we can do to make again this code more resilient these applications more resilient by how we handle it through that process of you know develop, you know through build to Cloud right Cloud.

I The term that the marketers have come up with for that but you know, it's interesting to me right because one of the things about devops one of the things about pipelines it was so compelling to an old security guy was the ability to actually just build this stuff in right and and have a lot of these testing Frameworks a lot of the structure in place in order to be, you know, exercising the code with security issues right before it ever hits the it's it's the, you know kind of hits the point on that front. So have you seen that and and what are some of the things that you guys are thinking about in terms of again kind of making that environment, you know, not just safe, but but also resilient Yeah, it's a great question. So I think there's a couple different components to that.

I like how Tanya started about talking about the misconception of Shifting left. A lot of times people say I shifted left what they mean is they took their traditional security tools connected into the pipeline and then someone is supposedly looking at the results or to accommend that Jeff made earlier that they want to move faster and then all of a sudden you see them turning those things off because they take a long time to run because they're not built to work within that environment. And so I think when you really talk about that shifting left, it's got to be about context aware scanning context aware results and what I mean by that is hey, I'm a developer.

I wrote 20 lines of code. I push that 20 lines of code in I shouldn't see a stack and Analysis result of the entire project as you see about the code I changed. And then to the the second part that's like how do you drive that that sense of urgency as part of that?

And I think that's where I go back to the if you realize developers don't wake up in the morning want to write vulnerabilities in their app. If you can present them the security results the way they can understand them and Italian Touch on that a little bit at the beginning about the importance of that. Then you begin to see that people are practically wanting to address the problems as they come up.

And then if you start to look at it from that point of view, I like the comment earlier that you made about the shifting right? I think you then take those good practices and you also move them the other direction as well enable the Ops Team. Though I will say the one thing that's been really interesting to me to watch and the industry and what we're doing at gitlab is like what is shift left actually mean and what does it mean to make the entire like cic pipeline secure and it does actually have to start the ID.

So like your comment was great. Right? So like what we've been trying to do is push those Securities results like in real time into the IDE because you don't have to worry about the code hitting the repository then by doing that you've actually thought about security all the way back to almost like the product manager level like I'm running a requirement.

What does that requirement look like? How does that now impact the developer? How's that impact the QA team the security team the Ops Team and then you begin to see that really much more secure pipeline.

I I will say the one comment Jeff about the dead packages. I think what's interesting about that is I don't think people are fully aware that they pull in all of these different packages dependencies these five percent of it. And then is that 5% the part that's vulnerable or is it the 95% Not touching and what does that mean?

And I think that's the really the kind of like the icing on the cake. If you can get the security to the developer and the idea and in that planning State and you get the visibility in the at the end if you get that information about what is actually vulnerable and there's a lot of really great dependency products, like gitlabs that can tell you those things right you begin to see what your true risk is, then you can have a true secure pipeline. I'm glad you started talking about context because I think it's critically important that we provide vulnerability information with full context.

And what I see is when you move really far left, you lose a lot of the context of what the application is doing how it works what systems it talks to on the back end which packages are used and which aren't and so on and so my concern about moving the analysis into the IDE is that you lose all that context and you end up with lots of false positives and lots of false negatives because you're only looking at a kind of a window into the code. If you do that analysis later in the pipeline when the app is fully assembled and you can actually see it run. You can provide really rich findings with lots of context and then I think you can report them back into the IDE.

So that developers can get that accuracy and that rich context in their findings, but doing the analysis with just in your super far left. And are just against the source code doesn't make a lot of sense to me. I think you're there is static models and dynamic analysis.

I agree with you on the dynamic analysis. You need more of the context of the app. But if it is a a SQL injection and it's not properly escaped that's not going to be a false positive that's going to tell the developer.

Hey, you forgot to escape this line, right? I agree with you that there's there's a need for context, but I think you have you have Baseline context on a stack and also scan that could identify those things. So they don't hit the project first and static analysis false positives on on so SQL injection all the time like 70% refills plus rate.

It's not I don't know you notice again. I think a lot of it gets back to I don't think that either of these are mutually exclusive obviously. Jeff has one point of view given what he's done for the last 20 years David has another point of view given what he does, you know every day and coming and seeing that and and I think that's okay.

Right. I mean, I think that the answer is at the end of the day we have to figure out a way to get the information to the developers with the context and the urgency so that they understand what they need to fix right because you know again, what is a critical one? What is you know, just something that we can put into the you know again feature list or defect list or what have you what needs to block a build right?

So that's kind of the thing right? I mean, you know, the point is Jeff when you get to a point where the whole thing is fully functioning and running in the environment. Is that when I block the build is there some way to block it earlier so that you know kind of I'm not spending all this time, you know with all this code, you know before that.

So yeah, I don't know that there's a right answer in a wrong answer but I think there are multiple point of views there. So what I want to do is To dig into this whole when can we break the build right? Because I work with a lot of companies and I'd say hey what circumstances do you break the bill?

Um, we don't what do you mean? They're like, wow, we've got note in what they're really saying without saying it is we've got no juice in the organization and the developers pretty much do whatever they want. And it's all about shipping code and it's not about security, right?

So how do you start to introduce that Concept in and by the way David and Jeff you guys are part your software companies, right? You work for software companies. So do you have a different perspective on you know, when you break the building you always break the building it's a critical one or you know again, I just love to kind of have that perspective too because I think that's part of it as well identifying the defect is one part of it.

What we're gonna do about it on the back end is certainly another part. And and John and Tanya, you know, you guys want to jump in, you know again be, you know be welcome to do that as well. You're throwing an idea.

Okay, then. I feel like if you want to be able to break the build you need buy-in with the software developers first and you need to run your tools in alerting mode and screen as many false positives as possible teach them how to use the tools so that they're the experts you're not always interjecting and then eventually when the results are really good then start blocking that's a better way to get buy-in. And also if you can I realize I talk about education and advocacy a lot.

So that's my favorite thing. But I worked out a bunch of places where they just didn't take the security of software even remotely seriously. They're like, we filled out the form we check the boxes and now we're done.

I'm like well actually and so I found that speaking and talking and getting buy-in and that might be a presentation about a cool new tool. You got it might be about something bad that happened. It might be about how you know, if we do this this way, it could be way faster and you can get out to prod on time.

Can we try doing this together? I find that breaking the Bills should be like way late on your instead of building up all those other things and Trust. I have a friend and whenever she shows up for the first time for a meeting.

She says I'm from security and I come in. Peace. And if you're like, when can I start breaking bills?

I'm gonna stop those jerks. You're not gonna have like the good thing and instead you're like come with your hands open and you're like I'm here in peace. I want to help you build more secure software, right like we can go a lot further, but I want to hear John's comment.

Well, I was going to say sort of I'll bet you any amount of money. That solar winds is breaking bills these days. But sort of like that you don't meet that resistance after the excrement hits the ventilator right?

Then the next time it's when let's what we're more careful. That's human nature. That's what I've seen the success that you think about in the early days.

The only way we could do security was added to the network. Nobody would let us get on the host. We couldn't do that at all.

And what what became the success stories then we're wearing there was integration between the knock network operations and security and they're using common tools because they're doing almost the same thing. The network guys are trying to measure Network performance the security guys you're trying to detect Network anomalies in the same is true on the host and these new apps. They need to build in Performance Management performance monitoring you all the things they need to do to get the app to run forget security and the commonality between what security wants to see is like 80% of the needs.

It doesn't mean forced to security tool on the developers, of course the developer tool on the security folks, but it turns out there's a bunch of tools on one side of the other that do both good enough for both and then Looking at a common view and then you don't you it is faster. If you build it, right the first time Jim Ralph at the board five million dollars that they would be able to build secure apps that Edna and shorten time the market and he he won that bet. And the five million, is that why he's retired now John exactly.

That's why. Hey real quick my take is you got to split this into two pieces. There's the backlog problem and then there's what it looks like after you've eliminated the backlog.

So if you go in there and start breaking the build and you've got a big backlog of vulnerabilities, it's gonna be hell. but you know, we've got new transparency requirements coming around testing. You're going to have to clean up your act.

You're gonna have to work off your backlog and once you do that. Then you're in a different situation then you know, you're the typical application project introduces two or three vulnerabilities a month. And if you give instant feedback that number can go down to less than one per month pretty quickly.

And if you're looking at less than one vulnerability a month breaking the builds, no big deal. But if you're if you've got 20 or 50 vulnerabilities on that project that you haven't ever closed and you're breaking the build all the time. That's a real problem.

So you got to look at the context and the maturity of the projects that you're working. Can I add one thing really briefly on top of what Jeff what you said is is awesome. I also feel like How fast you can go to prod?

Is part of this problem so a friend called me last week, and she said want to hear something funny Tanya. I just approved your merge request from 2021. And as a bug fix that I did I fix a vulnerability for I fixed a bunch of them and pushed them.

She's like I found like seven things that you did in 2021. It's still aren't pushed and she's like, I need to look for a new job Todd. Yeah.

And so I feel like if it takes over a year to push a vulner but like the depth fixed it but then you just can't get it to prod that's big part of the problem too. And so you don't want to be breaking bills. If you can't even get patches out and every single amount of time.

Yeah. Anyway, once again is is a fantastic segue to the fact that we can't solve all of our problems in the pipeline, right? We can't solve all of our problems by getting the developers to you know, develop secure code, right and Jeff.

I think when when you kind of talked about love for shell and then obviously the love for Jay issue that's kind of example number one, right? This is stuff or back. You know, again, we're a bunch of more experienced folks which is my synonym for old.

Um, you know, but Remember, you know kind of jet database, right? You know, what the hell did you database? net because jet database was pretty much everywhere and it was you know, software with vulnerable vulnerabilities everywhere, so You start to think about all right, if we can't kind of do everything from developer to deploy what are some of the things that we can do.

After right and you know, everybody talked about laugh and you know that the laugh vendors should send flowers every year to the PCI Council for requiring that you know way back when because that made a whole bunch of people a whole bunch of money. I didn't really impact the security the environment but not what standing a lot of people bought a lot of laughs coming out of that. So there are a whole bunch of different Technologies.

Now, we've got API security things that we got to worry about you got other different ways. I mean, you'll talk about harassment some of the other, you know, kind of options. We have that once something has largely been, you know, kind of run through that developer cycle.

We still want to kind of supplement it or or provide some compensating controls around this environment. So Jeff, I love you to start that discussion since you again based on your background and a lot of the the vulnerabilities you've seen over time. What can we do kind of before and what do we have to do kind of as as you know, a preventative defense maybe on the network is John.

It's Well, I think you're right on both counts. It's unfair to expect developers to write perfect code and also wafts haven't proven to be a very successful defense. I mean, they're really not realistic for modern apps that have you know, multiple pieces running in multiple different environments.

How do you put a firewall around that and it and it only works on the HTTP connections right? Not on the mq connections and the other serverless connections and all the back end things that are going on. So we need some some different solutions there.

To help developers. I think I asked is the right thing it monitors applications from within while they're running. It's like sort of super desk, but from inside the running application and you don't have to do any real security testing.

You can just find vulnerabilities by doing normal QA activities. So that really changes the math on on finding accurate vulnerabilities quickly. But after an application is deployed.

I think the first thing everyone should do is deploy runtime protection. There's a technology that hardens modern application Stacks. So if you think about your modern stack, it's got all kinds of dangerous stuff in it, like, you know SQL connections and expression language evaluators and unsafe dish realization engines and so on and rasp ads protections around those dangerous capabilities to make it difficult for developers to use them incorrectly.

So it's an easy way. We did it an experiment at a large Financial organization on 1600 of their vulnerabilities we check. Hey, what is what does rasp prevent from being exploited?

It was 95% of them. So only 86 vulnerabilities were left unprotected after adding rest. We thought that was really exciting.

And so, you know, that's sort of the first thing I recommend folks to do the second thing. I think for for things like Library security. I think having a tool that can measure like an is or a runtime SCA tool that can measure how libraries are actually used what they do and what they connect with is really powerful that'll allow you to eliminate all the libraries that aren't actually ever loaded or used at runtime and you don't have to worry about vulnerabilities in those dead libraries.

You can just focus on the libraries that are actually running So those two things can bring you a long way towards securing your your pipelines and environments. Sending news Tanya. Go ahead.

Yeah, I don't have a dissenting view. I have an additional more things that we can do and I might be jumping ahead of it. But I wish that we could work with the framework makers to make things more secure.

So I'm from Canada. A you probably hear me say a boot at some point. And so I just Process.

So are you I used to live in Ottawa. We had an OAS chapter and we would have our monthly meetings always in the big Shopify building. So I got to hang out with the Shopify absec team which was really cool every month for like years and so shopify's built in Ruby and they have this huge bug Bounty program.

And so what they would do is they would pay the public to find bugs and any of the ruby gems so like like a new get package but for Ruby anyway, and so they spent like a million dollars in bug bounties to prepare the gems for everyone across the planet and like Canadians were weird and we help them pitch in and we're very polite almost offensively. So but like if we started changing the way we do open source, so like it's cool if you want to build on this framework, but if you make you know, five million dollars of profit or more you need to, you know, contribute five fixes or something so that then if we all start contributing to these Frameworks and to the components from within the frame, I think we could do a lot better. We can't just expect like one Canadian company to fix an entire framework, right, but I think that if I don't know if this is a legislation sure my experience because I spent eight years on the Java servlet spec team.

and my role was to try to get security improvements into the Java servlet framework and In those eight years. I managed to get like two things done. I think you got secure flag put on cookies, and I got the ability to prevent new lines from being put into headers.

That's it. I had a list of about 40 things that really needed to be done. And it's it it's insanely difficult to do this even for one framework and there are hundreds of Frameworks.

And then we have to kind of cross multiply those with all the different app servers and all the different runtime environments and all the versions of those things. It is an absolutely intractable problem to try to go back to all those places and try to Stamp Out vulnerabilities that way So you just don't see yeah, that sucks. Now, I'm depressed Jeff.

eight years That's a really long time and that's quite a lot of dedication. It feels like when I was trying to get folks to agree on security metrics, you know, probably about 20 years ago. I think I had the same the same experience Jeff.

I banged my head against the wall 50 times and you know, then gave up. Um, anyways about other, you know, kind of Alternatives that we have. Yeah, I think I'd fill off of what Tonya was saying.

I think that one thing that works. Well is the Bound by Grammy programs. The other thing that I've seen have a lot more likes to it.

The people supporting is the more you adopt open source, the more you feel responsible for it. And so if you are using it open source project and you know, there's something there fix it contributed back because now you're not just helping solve it for you. You're helping solve it for a bunch of people and I think that that's really a good Community feel to it as you do that.

Yeah, so we got a couple of questions here and I think one is is pretty relevant to you know, this discussion, right if I have a solution for code analysis. Should I include other types of security testing within my application so I don't know which one of you guys want to say over sounding? Yes, but say over sounding yes.

Yes, yes show that they're doing it. What's that John and require that all your coach suppliers show that they're doing it as well? Yes, so we will get to that moment Tito but okay, it's magic.

It's not just like the goal is not to run all the different kinds of tools. The goal is to verify that your security defenses are correct and present and effective and you should use the smartest technique for each one of those defenses like, you know, if you want to check for hard coded passwords great, you know scan your repo and look for them with a tool that does that but if you want to test for injection flaws you want to use I ask because it's so much better at data flow analysis. And so like, you know, the you shouldn't do both techniques for one kind of problem because it's wasteful you'll end up with with a bad set of results should do the smart thing for each kind of problem.

Yeah. No, I I also think that I try to look at applications from from different viewpoints. So you to like Jeff made I ask which can do like an interactive like well, it's running but it can also see the code.

I think it's really important that we do something to analyze the written code that we verify that the third party components were using the way we're using them is safe to do so slash they are safe to use because a lot of them you have something that has tons of bugs in it and it's in your app, but you're not calling that part so you're still okay? And then something to verify the way that it works that there's not business logic laws that there's not implementations in the code from an interactive perspective. Like when you're using it, is it working properly and then on top of that more protections like numerous other layers of Defense depending upon how sensitive the thing is that you're doing because this is the thing that allow us don't talk about is that if you are doing character terrorism stuff for government, you have very very sensitive systems.

But a lot of us are Building Systems that really aren't that sensitive and like if it only brings in a million dollars of Revenue per year, you're not gonna spend 500,000 securing that system unless it's gonna save lives, right? So we're unless it's good Nation if yes actually but yes, that's one of the things that you know, I want to kind of wrap up with because you know one it's it's a nice way to kind of put a bow around a lot of the very discussions that we've had here, but it's also, you know kind of really indicative. The challenge that we all face, right and that's again if we don't solve the problem somebody's going to come in and tell us we have to solve the problem.

Right? So, you know, the cybersecurity Mandate that came out of the US beds last year mandated, you know, s bombs right software billing materials. We already talked about the fact that that's you know, just giving you a list you could you could build a car you could build a bomb with you know, the little bomb right with those kind of things.

So how useful is that but again, where are we just you mentioned something about software accountability and making sure you know, the folks have to deal with their vulnerabilities, you know, John you've been front and center in a lot of these regulatory discussions, you know through the years. I'd love to get your perspective on what's realistic. What can we actually do in order to make that happen orders are just fade a complete that you know what we're gonna get a bunch of Regulation that nobody really understands how it works because nobody's figured out.

To solve the problem yet. Well, if you're realistic about it, we all know the Holy Grail is making software sellers liable for their products. Right if my Subaru has a fault.

I don't fix it. I'm Subaru fixes it right where you were used to that in the physical world. And unfortunately, the IT industry is escaped that so we're not going to see that in any of our lifetimes right liability tied to that Beyond but before we get to that think about when you go out to dinner, You can eat chicken marsala in thousands of restaurants.

It's made differently every time different Cooks different recipes different ingredients coming from thousands of different suppliers and rarely do you drop dead right? But if occasionally somebody gets food poisoning and and drops dead to me the software industry is very much like the food and restaurant industry. We do have some checks and balances in the food industry.

There's FDA. You can't leave mayonnaise out in the sun too long and your refrigerator has to work and it can't be rats running around in the kitchen or your restaurant get shut down. And so we're gonna have to get to the point where some of those things happen before the any of that happens though.

I think there are some near-term things that the vitaministration has proposed. Let me give you the one major success story. I can think of of government regulation having a meaningful impact on security when when the browsers first got started when Netscape was written the bad guys were skimming passwords at the isps on the network.

They were stealing all the logins are in the clear. so Netscape came up with SSL essentially right the base and they did a crappy job of building it they did crypto badly and it got found out quickly and it had to get updated and it led to the development of the Phipps 140-1 spec by nist for how to do crypto right more importantly. I wish I could I'd tried to go back to find out who did this in the US government OMB said any government agency buying crypto.

The provider has to demonstrate it was tested for compliance with fixed 140. That caused Microsoft and Netscape and ultimately Mozilla and the other browser guy and everybody was building SSL in they have to do that to sell to the biggest pocketbook in the land in the world the US government essentially right and that drove an increase in thing. So now we do see in the executive order 1408 or whatever about 28, whatever it is, finally them saying the same thing we're gonna use the government's buying power to drive and a great example of where it's working is fedramp the cloud services that the government buys that are more secure that the Google's and apples and Amazon AWS isn't all are doing things extra that then get baked into their normal commercial offerings to me.

That's the biggest thing is how do we address the pocketbooks of the people selling software to make it more expensive for them to fail or more lucrative to them to sell a more secure product and we're just starting to see finally some movement in that direction. That's right. And and I think that's a great way to wrap things up John.

So thank you for that. I want to thank everybody on the panel Jeff Williams David DeSanto Tonya. Junka, of course, John Pescatore.

There's so much in so many different layers to software security that we've got to continue to peel back obviously textural research. It's a major part of our coverage given where we sit in between devops and Cloud native and security. So that's going to be something we're going to be doing, you know throughout the year.

But again, I want to thank everybody for their time today and we will see you later on.