Omkhar Arasaratnam & Brian Behlendorf, OpenSSF | OSS North America 2023

This is Techstrong tv. Hello, and welcome back to the Open Source Summit in beautiful Vancouver, British Columbia. We're talking about open source security.

We have the new general manager for the Open Source Security Foundation, m Carre NAPman. And then we have Brian Bell Endorf, who is the CTO now, I guess. Yep.

Is that how we're gonna do this? All right. Awesome guys, welcome the show.

Thank you for having us. Thanks. There's been a lot going on in this little foundation of yours that just got started not too long ago.

We got a new gm, you got a bunch of projects updates, you got new members, so give us the highlights. So under Brian's leadership, there's been some great progress over the last couple years. I'm really looking forward to working with the foundation, all of our members in the community with bringing better, more secure practices to our open source foundation, and ensuring that we've got a way to help make our software a little more secure for the better of the, for the good of the public.

Uh, Brian, did you want to cover some of the highlights under your leadership? Well, first of all, let me say, you know, um, we've, we've been building an amazing team at the Open ssf, but I also realize, look, there's things I know really well. I've been part of the open source movement for 30 years, really.

I, I, and there's other things that I've not been a part of as much, which is enterprise security, right? Uh, the lingo, the, the terminology, the history. Uh, I've been a quick study and certainly as it applies to open source code, it's been, it's been, you know, a great, a great educational experience.

But as we were building the team, I realized I really needed someone, uh, to work with me as a peer who really understood, I, uh, the history of application security, knew how to talk to executives about risk management, knew how to build out infrastructure, uh, and where the open source code could fit into that. And just as I've been a quick study on security Kar, certainly, no, no, no, uh uh, no, no. Stranger to open source code as well.

He's been, who was a contributor to, um, power PC Debian, uh, 20 years ago. Gen two, gen two, I'm sorry, gen two, uh, 20, uh, 20 years ago. Uh, so like, basically this is a partnership that is really gonna be about executing on both the big plans that we have, uh, around securing open source software in the software supply chain, uh, at a, at a business level, working with our stakeholders, making an impact, you know, to boardrooms and the like, as well as on the ground, the technology, identifying the gaps, the missing pieces, and pulling all the tools together to come up with something that is cohesively an answer to addressing many of the security challenges and, and software.

It Takes a big person to realize that they've reached their level of incompetence. Is that where you're going with this? Basically, actually three times in my career, I stumbled into starting a thing, or like being a part of, like, growing a thing and then been very great, very much understood.

Like, you need to build it. It takes a village, right? It takes a, a lot of people with different strengths, and I really wanted somebody in that background and also somebody to, to do the kinds of things.

You know, let me focus on tech, uh, let you know, focus on, uh, stakeholder man. He could focus on stakeholder management. I could focus on kinda some of the nerdy thing, you know, frankly, like he can focus on the nerdy things too.

And I'm certainly gonna also talk to the business types. So we're gonna figure out how this relationship works, but, um, I'm really excited to have, uh, as a, as a partner in this effort, uh, Omkar and, and, and, and move forward with that. You've had some new companies join, I think SAP is one.

Um, what is in it for a company to join this particular foundation? Why should they be looking at this in particular? I mean, we all know it's an issue, but what motivates them to actually want to contribute?

There's a couple ways that we like to think about this. Um, the first is that security doesn't exist in isolation. We can't hermetically seal a company and say, I am secure unto myself.

And especially when we think about the prevalence of open source software, as it's used by industry today, if we don't collectively ban to address security concerns upstream, we're all gonna have to do hand-to-hand combat. Uh, the analog I'd used earlier today was want to get outta swatting mosquitoes. And that's what it feels like a lot.

We're trying to address some of these problems that aren't being properly addressed upstream due to capacity. As I think about our stakeholders, they fall into four different categories. The folks that produce software, our friends at Google and Microsoft, and a number of the other members of our board, the folks that consume software, you may do your own development, but you're relying on a technology company like our friends in the banks, or our friends in other industries.

Then there's also the public sector. So a large amount of addressing this comes from support and cooperation with governments, both in the us overseas in Europe, Canada, across the world. And last but not least, this is open source, our community, and being able to positively affect outcomes within the community.

So when it comes to, to get back to your question, the benefit that companies get is being able to collaborate on really assisting with securing our software, our open source software for the greater public good. And there's benefits abound. Brian, did you want to add anything?

Uh, You know, it's been great to see the support of organizations like Google and Microsoft who came in, uh, not just the, they've been solid participants in so many of our projects, but they also recently came in with two and a half million dollars in support, further support for the Alpha Omega project, which, uh, has gone from strength to strength. And we can talk more about that in a bit. But I'm also really happy to see support from small and mid-size companies.

I think of a company like C who are building tools, uh, to analyze and understand the software supply chain, small little startup that they're, uh, one of their, one of their technologists, Michael Lieberman has been really active in defining the salsa specification. Uh, and, and, and it really takes that whole, the whole spectrum of types of companies out there. We also have a tremendous number of end user organizations.

Now, we recently added Lockheed Martin, for example, as a member of the project, and having them in there participating and, and benefiting from it alongside banks, alongside other very traditional companies, really shows that this is more than just like a cloud problem or an infrastructure or, or a tools maker problem, but even really about the end users, the consumers of all this technology, they want secure software as well. Then a lot of great ideas are coming out of that side of the system. Well, the maintainers been reacting to all of this because there are diverse bunch of folks, and some of them might be, we don't need any help from anybody, we're good.

And others may say that, you know, we'll take all the help we can get. So it all depends what the spectrum is. But as you talk to folks, what are you hearing from them?

Do you mean maintainers of the projects at the open ssf? No, The open, just open service software maintainers themselves. Well, The Linux Foundation, you know, regularly runs research and, and does polling of, of developers in particular.

Uh, and we really want to understand who are those most critical open source developers and what drives them and, and the like. And to a t every one of them says they wish they had more time from their employer to be able to work an open source code to do not just the feature work, but also pay off the technical debt and the security work and all that kind of stuff. The second thing is they, that they say is they want the same kind of developer tools, uh, uh, that they, uh, when they work on open source code that they do when they work on their company's internal proprietary code, because that extra tooling gives 'em a degree of infrastructure and a degree of rigor and the degree of templatization that makes it much easier and much more productive for them than the stuff they have to deal with in the open source side.

So one of the things we've inculcated from that is a need to make sure that as we come up with tools for signing artifacts like SIG store and, and, and salsa, which is for tracking providence in the software supply chain, they're only successful to the degree that you can get them embedded inside the tool chains of the world and really uplift the open source tooling to, by default lead to better secure software, better, more secure software as, as, as, as a product of using those tools. So, uh, it was really great to see, for example, GitHub recently announced that they had taken these two pieces and some other things and woven them into support in the NPM package ecosystem for Providence traceability, uh, NPM dash, dash providence, uh, uh, is now a flag. It's all opt in.

And so I'm really hoping that we can turn it to an opt out kind of default kind of mode so that the NPM ecosystem can be much more traceable from source code to final delivered product. But that's a nice example, an illustration of where our, our members, having the right members in place working on the right technologies, but getting this into the tool systems can meet some of what the maintainers are asking for. And back to your original question, no one wants to lift a finger to do an extra amount of security work.

What they want is the tools to do the work for them and for the ecosystem as a whole to bend towards more secure software as a result. I think when you guys first started, you made a call for a, I think it was an estimated 150 million in funding or something like that. Is that enough?

It seems like a small number compared to like the billions of dollars that we spend on software. So We're always willing to get more, I think is the right way to begin that. But to be realistic about it, there's also a planning and execution headwind there.

Well, not headwind, but overhead. So with any amount of money in order to ensure, especially with donations from our generous board of directors and member companies, it's important to be judicious with how that's being allocated, how the progress is being tracked and the impact that we're making. Not everything can be solved with money.

Sometimes there's fundamental structural things that we need to agree on in terms of data formats, things like s bonds, that's not gonna be solved with a check. So I agree that this is not going to be the last check that's cut in aid of securing software. However, we felt, and Brian, you were there at the time, so I'd love your input as well, but we felt that was a good start.

It was something that was manageable and it was something that we could allocate appropriately to the community in order to secure software for the greater public good. The, the mobilization plan, which is about a year old at this point, um, was devised at a time in kind of the aftermath of the log for j uh, log for Shell, uh, vulnerability at a time when government was looking out there going, well, who's actually gearing up to solve these problems that we've identified as root causes of, of that, of that, uh, the whole disruption? And they turned to us and they said, what are you all doing?

And we've got, you know, we said, we've got all these interesting projects and efforts. And they said, well, really, what does it take, what will it take to apply those to solving the problem? And so for us, that document, which we devised in our community in Rapid, like in basically a two week window with, uh, a whole bunch of experts convened really rapidly around 10 different streams, uh, was intended to represent at that moment in time a true north, right?

Here's where we could go, should the resources apply, here's the kind of impact we could have for what, by some measures, there's a lot of money, certainly more than I've ever had on my personal budget, um, but in some ways is a drop in the bucket compared to the impact of even one of those vulnerabilities, like a log for shell. So we, we did that as a, as an example, like here's what's possible. It will be updated this year to represent a new set of true Norths, if you will, a new set of ways to take what we're already doing and apply it to those big problems and how they'll solve it, but also identify, here's some new things to start working on, such as where does artificial intelligence land in that plan, right?

This year we should be talking about it 12 months ago, we could be forgiven for not talking about it, right? G p t hadn't shown up on the, on the, on the map. So, so it serves as basically that top-down map and that top-down kind of view of what's possible, but really all the substance and all the hard work comes bottoms up through the maintainers and through the folks working on standards and, and all those that, that have to come together to, to solve those problems.

Now you mentioned the Alpha Omega project earlier, but I'm not sure everybody knows exactly what that is. So walk us through what that project is, what its scope is. Okay, Sure.

So Alpha Omega was like many of the things that we started right when, uh, we got funding for, for, uh, for the Open ssf was based off of a white paper written by Michael skt at Microsoft that said, you know, there's these two interesting things we could really do to help, uh, raise the floor on how security is accomplished in the, in the open source landscape. One of them is go and, and provide directed funding to the most important open source projects, and in some cases, foundations to help them get better at managing security across their foundations. Like the policies and the processes that the Python Foundation or the Rust Foundation or Eclipse or many of these other kinds of orgs could use.

A, could use an uplift, right? It could use somebody helping them get over a hump to be able to demonstrate the value of this work, and then hopefully go back to their own stakeholders and say, now that we've got these engines running, let's continue to get that work, uh, funded as a, as a core part of that project, right? So intended to help Boots bootstrap, uh, a lot of stuff that had not been done internally by some open source projects.

And so we've been very successful at that front. We've distributed over three and a half million dollars so far in grants to six different organizations. We have stuff that we've approved and not yet announced that we'll be getting the word out very soon.

Um, that'll address memory safety in a number of other ways, but it's about capacity building and just changing a bit of the culture in these important projects. The second thing Alpha Omega does is we're set standing up infrastructure to go and scan for vulnerabilities in the top. Currently the target is 10,000, but we're looking at expanding that even more widely, the top 10,000 open source projects to go and look for the kinds of vulnerabilities you'll find in a whole lot of them, right?

Really simple kinds of bugs that we know. We find if we go and scratch that, that surface, we will find that in a lot of places, go proactively find them and submit poll requests to try to, in an automated fashion, get those fixed, uh, by thousands of projects at a time, right? So that is an ambitious goal.

It's based off of some research done by somebody we hired named, uh, Michael Lek, uh, to go, uh, uh, Jonathan Lek, sorry, uh, to go and automate these processes. We're really excited about it, and I think that'll help, again, raise the floor on security for, uh, the most important open source project Because there's a lot of common mistakes that everybody makes, and there are, there you go. Do we need a SWAT team to just go handle all these issues when there's zero day vulnerability shows up and is there's, you know, can, can I just call you up and say, you know, Batman, come fix this.

You can, I'm not sure I'm gonna be very helpful though. Jokes aside, uh, the efforts that we've had thus far have been more around securing the software by construction. So coming up with methods of addressing in large swats of open source, some of these common missteps that we've seen developers make unintentionally be they memory safety issues off by one issues, whatever they may be.

I think eventually as we get our arms around that, there's certainly the opportunity, and I think it's something contemplated in the mobilization plan as well as to how we do come up with, in a product company, we would call them a P cert team that's focused on product security. And to take perhaps a, an example from the not so recent past with the kernel, you know, how something like Specter Meltdown was addressed, how the right people were assembled in order to address this, both from the hardware community, the kernel community, the providers themselves, in order to ensure that there was an appropriate response, but also that confidentiality was maintained as we went through this due to the severity of the issue. So I, I'd like to get there, Brian, maybe you can speak to some of the thoughts around that.

Uh, part of me joining, well, You know, there's lots of different kinds of open source projects out there. Some of them have emergency response teams and security teams that are reasonably well advised and resourced when there's a bug in the Linux kernel that requires us all to coordinate that gets done. But so much of the long tail of open source projects have never been through a major vulnerability disclosure process, have never had to deal with the cbe, and they don't know who to turn to for help.

Uh, even some of the foundations could use some help on this front. And so a, uh, an emergency response team, uh, coordination center, that is something that was anticipated in the mobilization plan. We've started to put some plans, more concrete plans around that for what it would take to coordinate basically a, a super friends kind of like, you know, DC superheroes, kind of like, kind of like collection.

Now you've, we gotta be careful about how you set those up and you've gotta have the right kind of volunteers in there. And, but, but doing it right, it's not gonna take a lot of money. It's gonna take a bit, but not a lot of money to go and get a tremendous amount of impact, uh, in, in reducing the, the kind of like chaos that can come from uncoordinated vulnerability disclosure.

Do we fundamentally to address all this, just need everybody to rewrite everything in rust or some other memory safe language, or what's the deal? I wish it was that simple. Um, we should start there, but I wish it was that simple.

Personally as a identify, as a software engineer that's been doing security for 20 years, and the way that I like to think about it is we need to ensure that our design patterns, our tools, our languages secure by default. I don't want to have to spend extra calories or cognitive overhead in order to think about making a thing secure. It should just be secure.

And one example I've used in the past, there was this anecdote about a factory that was producing a particular part of an automobile. They ran into a quality control issue where a part was being put in, in the wrong orientation. So they added a QC engineer, and then they added another one and another one and another one, and eventually they plateaued in terms of the quality that they were able to achieve through that method.

But when they retooled the part, so it could only fit in, in the correct orientation, that was how they fixed the problem. I think applying that similar thinking to how we're addressing security rather than trying to qc stuff in alone or trying to incident respond to things alone, if we can just make it secure by construction and design, he said simply, we avoid so many of these things in a very efficient way, and by that we really increase the security guarantees that we can provide the public. Right?

Yeah. I, I, um, I hear a lot of the advocacy for, for memory safe languages really ranks the, the c and c plus plus folks out there who are like, you know, or, but, uh, but, you know, go and rest and, and even Java as a memory safe language, right? Uh, are are actually pretty compelling for eliminating a whole category of vulnerabilities.

But look, log for J was written in Java, right? Right. And it wasn't a memory issue so much as it was about parsing user, user contributed input for formats, ah, formats.

So like, there're bugs of all sort. Uh, there's, uh, it's been great to see the White House surprisingly prioritize memory safe languages in its cybersecurity strategy as a potential approach to eliminating whole categories of vulnerabilities and that, and we think for a lot of the, you know, baseline libraries that are out there, like Russell's, which is a alternative to open SSL and is gaining a lot more functionality, and eventually it'll be kind of a drop in replacement for open SSL is a really promising area of direction, uh, area of focus. Uh, the Proximo project is looking at memory safe replacements for NTP D and these other forgotten, but still essential components to, uh, the, the how the internet works, right?

If something goes wrong there, we run out of sync on time and, and bad things happen that are hard to predict. Um, uh, so like, there are these interesting targets. I don't think we're ever gonna rewrite everything in rust.

We can barely rewrite all the jo all the COBAL code that we have into something more modern, right? So I don't think that's essential, nor is it is is it sufficient? Um, uh, we've gotta be thinking about process, we've gotta be thinking about scanning.

We've gotta be thinking about how do we make sure every line of code has more than one person responsible for it and looking at it, right? All these different factors need to come together in the tooling and in dashboards so that we can go, aha, there's something that's red down in that corner that's under-resourced, it's critical to everything. Maybe we should get a third party audit for it and take it from yellow, from red to yellow so that our whole risk comes down.

And that's, that's where I think it's a, there's a really holistic picture around what we've gotta do, of which memory safe languages are certainly helpful by just a piece. Will AI save us from ourselves? Because, you know, I can now discover the vulnerability.

I can then be shown, here's the line of code that you should fix, and then here's the actual fix, and all I gotta do is push the button and install it. Is it gonna ever get that simple? I hope so.

I don't think we're there just yet. I don't think we're there just yet. And I think similar to how in the very early days we were learning how to develop code that would scale well and be secure on the internet.

We're in the very early days of generative ai. And I hope we get to the point that it's that simple and that we can really leverage large language models in order to automate some of the toil that, or some of the long tail that we're addressing in Omega. I don't think we're there yet.

Um, but I do hope that that's some, that's an opportunity that we can look at in the future. I, I think, I think there's an opportunity for, for it on the defensive side, but I am very worried about new kinds of attacks that AI can enable. So much of what we do in the open source world still depends upon trust between individuals, trust, you know, built at conferences like this trust built over Zoom calls with other maintainers.

If, you know, large language models make it possible to have fake developers, uh, you know, inserting themselves into your community or otherwise, you know, like a lot of, a lot of the burden of being an open source maintainer is in bug triage. You know, if you wanted to flood a project with a lot of fake bug reports that you generate through, I mean, there's a lot of like dark, dark directions that this stuff could take. And so, or if I could, if I could be the, uh, slightly tinfoil hat wearing security guy for a moment, what if you poison the model and you start spitting out code that's got a bunch of memory safety issues, right?

It's, we used to call that the watering home attack, but it's, it would be the perfect factor, right? Yeah. So what type of people are you looking for to help volunteer for this foundation?

Is it security people? Do you need developers, um, kids that can contribute or has it gotta be somebody who works for Google? What's the spectrum of folks that you're looking for?

We, we love support from all of our board member companies and we've got great contributions for them, but the answer is everybody can contribute. There is a number of different work groups from policy to best practices to tooling to anyone can contribute because it's truly democratized. Just because you don't know software engineering or just because you don't know policy doesn't preclude you from helping.

We need hands, we need people that are interested. This is a community effort, like all open source efforts and we would love anybody to help come, contribute, come learn. Education's a huge work stream as well.

And I think that really allows anybody that has the desire to help us be more secure for the public, good to contribute. O'Brien, what are any hotspots? The Only thing I'll add is, you know, my first message to folks would be, if you're not already familiar with what we offer, come to the website.

But also if you're a developer of any sort, even if it's like visual basic macros, you know, like come and take the, uh, secure software development fundamentals course. This is not super advanced. This is something that anybody writing code should take that would help them avoid, uh, common mistakes that tend to lead to vulnerabilities, like trusting user contributed input, um, and or or least, you know, for format strings, right?

Like, like, here are the patterns that you can learn to avoid even writing that stuff in the first place. Uh, there's other one page sides to this and that, how to be a better consumer of open source code. There's things that, that we think can speak to people at any level, uh, uh, and if they start with that, uh, and if 1% of those people who benefit from them find a bug, uh, find a way it could be improved, come in and engage in a conversation or have an idea for something they've built or their company is built that wants to come in, that's golden for me.

But it has to start with this like, intrinsic motivator of they've benefited from it, they're familiar with it, they've worked it into their company's practices. That's, that's the real milestone for us. Uh, so that's the, the, the door is open completely to anybody to, to do that.

At the end of the day, are we trying to retrofit a set of DevSecOps workflows into the development of open source software? And we're really asking people to retrain themselves or rethink how they build software. Because in my experience, most developers when they were trained, security was an elective and they never took it.

So, you know, how far back do we gotta go? I agree with you. It's not something that was historically taught, nor were there standard practices around how to address that.

But if we think back to the late nineties, early two thousands when the internet started taking off and we really started getting into some fundamental new patterns around distributed system design, and we started to write code in different ways and we started to adopt different patterns. I think that's the same kind of apex that we're at right now. So rather than simply continuing to do things the way we did in the past, resulting in even more instant response teams and even more Omega projects and things of that nature, this is what our education team seeks to solve with some of the courseware that Brian's referred to.

But this is also areas we're looking to partner with public sector with higher education to help get some of that to those students that are now learning how to build big, powerful distributed systems who haven't yet had their first job in industry. It's just like, I hope to make my children more secure by teaching them how to cross the street safely. I don't intend on being there every time they cross the street.

Nothing to add. That's great. What is the role of the government in all of this?

Because it seems like we're doing this within the industry. There's a lot of calls for more accountability, more liability. Some people are saying, you know, if you write a line in code, you have to own it for the rest of your life.

What's, what's the right balance here, Brad? I think you got some point of views. Well, yeah, So I do think, look, open source software has always, uh, benefited from actually the same thing the whole software industry has traditionally had in their end user agreements, which is, this code is used at your own risk.

You know, if you lose a bunch of money because there is an error in your spreadsheet, you know, Microsoft's not liable and for, you know, just cuz you used Excel, right? That's a basic principle that we've all kind of like taken for granted. Um, and open source licenses in particular say this is use at your own risk if you happen to use this in a nuclear power plant and cause an accident, you know, because there is a bug in it.

It's not on, it's not on the open source developer. And I think that's the right thing, but I want, I think actually society's response to the log for shell breach to, to the other things that have happened, the log for shell vulnerability, other things that have happened in the last year have caused them to go, you know, maybe this Silicon Valley, maybe the, the open source community can no longer, you know, move fast and break things. Maybe it needs to think about owning at least a little bit of the results of their work.

And so you do see very much in the European Union and their, in the regulatory moves that they're considering. And even in the White House cybersecurity policy, a bit of a hint that, you know, we need to look for potentially some of that liability being born by the parties that can bear it right by the publishers of code, or at least the companies that are reselling this reselling, uh, products or services based on this code. Now we think it would be really a bad idea to put that liability upon open source developers who are for the sake of publishing something to GitHub, you know, are making it available to the world.

Much as I would have an orange tree in my front yard and with a sign that says free oranges, right? Uh, and if you happen to be allergic to oranges and got sick, I mean, like where does the liability for that lie in Europe? It would be on me.

So we are gonna see a shift in this. Um, it does sound like though the regulators are realizing that there is a standard for due diligence that should be fairly low, should be fairly achievable, should be something that is, is something that developers could hit out of the box that would give them a little bit of an exemption from that liability. Uh, and so in the United States, the, the, what was in that cyber, the, the cybersecurity, uh, strategy, uh, was basically attaching that to the commercial transaction at the tail end.

If you're a red hat, you'll, you know, there's a bit of that liability to be expected, but if you're somebody writing a thing that's included into Red Hat, you shouldn't bear any liability for that in Europe. They do want to kind of make everybody like accountable and, and, and liable in that. So we'll see which way that goes.

And it could be a major clash in Europe between how the open source community works and how the regulators think that it's supposed to work. And we'll see what happens there. Um, my hope is that we come up with tools and, and processes and standards in the open ssf that to whatever degree these start to become more frequently asked for.

Uh, and, and things that open, properly run open source projects should do, just like we've done on the legal side, just like we've done with dcos and sla, uh, SLAs and things like that, that it becomes a thing that's achievable in a, in a very scalable kind of way. Um, so that end users can be reassured that at least the due diligence has been done to try to provide secure software and maybe even make choices about what software they use based on that. The only thing that I would add to that would be just to, I think it's important to understand where the impetus comes from, and the impetus comes from scenarios in which we have things like iot devices that don't have any committed updates that are running vulnerable versions of software and have been used for things like distributed denial service attacks.

So there's certainly a, we can't swing to either extreme. There has to be homeostasis in the middle. And I look forward to working with both our private sector and public sector partners with finding where that homeostasis exists.

So are we having something that feels like the equivalent of a Ralph Nader moment here, where we're unsafe at any speed and you know, are we building software too fast and we need to slow down? Or is there another way to think about this? So in my opinion, I believe that none of this, none of this was done with malice or disregard.

I think as we are evolving and as the discipline of software engineering's evolved, we're discovering things about how people use software and misuse software. And we're trying to come up with the right patterns, techniques and tools to ensure that the misuse, the impact of the misuse is minimized and that the right path, the correct path for using the software is maintained. I don't think it's that extreme.

I think we we're making incremental progress and we will continue to make incremental pro progress. I don't think, I think, excuse me, we're past the days in which we could disregard whether a particular bit of software is using proper input sanitization or whether, uh, don't worry about memory safety, don't worry about counting how many bites are left in that buffer. We're past that and we need to be thoughtful about that if the software is going to perform at scale with the kind of surface area we have these days.

The other thing I'll note in the nineties when I began working, there was a token ring network, there was a squid proxy and I had no direct access to the internet through my desktop. PC light bulbs now have IP addresses, so the surface area is fundamentally different now as well. Yeah, and nothing, Nothing about that.

Gentlemen, thanks for coming by the show. It's great to have you guys. I think what we learned from all of this is that software hygiene matters, you need to start paying more attention to it because if you don't, your software might stink a little.

All right, Thanks so much. All right. We'll be back in a minute.