Advancing Embedded Open Source with The Linux Foundation’s Kate Stewart at OSS Seattle 2024

This is Textron tv. Hey everyone. We're here at the Open Source Summit, Seattle 2024, enjoying a great show.

It's really been, uh, a lot of energy, a lot of passion, a lot of cross polarization about different kinds of open source project, which is of course what brings us all together here. So I have a pleasure being joined by Kate Stewart with the Linux Foundation. Yes.

Our first time talking. It's great to talk with You. Thank you very much for having me.

Yeah. I know you're, I know you're getting ready to zoom into a panel Here. That's right.

Yeah. So, uh, if we can kind of capture a little bit of, um, sure. Knowledge and passion.

Tell us about what you do at the Lenox Fund. So, um, my role at the Lenox Foundation is focusing on making sure that embedded open source becomes dependable. And so what I'm working on is how we can improve the transparency of open source and then especially on the embedded side, how can we actually start to get it so that when people are designing with open source for things that maybe have safety elements or security elements, that we actually can make it easy for people to automate and actually do the right level analysis for being able to trust what they're working with.

Can you say a little bit about embedded software for people who may not work in that space? What are some of the unique characteristics of it? So, embedded software is usually software defined for a specific purpose.

Uh, things like your car, things like, um, you know, a blood, you know, a a, a device like a array. Blood, blood pressure. Satellite or satellite.

Yeah. So basically the resource constrained usually. Mm-Hmm.

Power is a lot more important in those spaces. You know, power's important in big server farms, don't get me wrong. Is Footprint still a, A big consider Footprint is very much a consideration because, um, Linux doesn't get smaller than about, you know, just 20 meg, you know, 10 to Meg, something that range right now.

And when you're dealing with a little sensor that's just basically looking at the temperature or something, or altimeter or things like that, you're not gonna wanna put that much memory on it. Mm-Hmm. 'cause that consumes power.

Exactly. And so, you know, ze, one of the projects I work a lot with is called Zephyr. And Zephyr is pretty much been designed for resource constrained devices, or lettuce is too big.

And that's in our tagline Where Linux is too big. Too Big. Yep.

Where Lenox is too big, where We love Lenox, but weight Is too big. And I love Linux too, don't get me wrong. I've been very active on a few other Linux related projects as well.

Um, and I'll happily talk about those too. But Zephyr is, you know, 10 K and up. Wow.

What a difference. And yeah. And so that's, So it's not a even more slim down Linux, It's all in No, it's completely its own new code base, everything else.

And it's been getting a pretty good community of building up over, over Simon. It's been quietly building over time. Mm-Hmm.

Um, and it looks very familiar to anyone who's worked with embedded Linux. It's got, like, it's got k config, it's got, um, the device, it's got device tree, and so you can just put in exactly what you need. And so that is attractive to a lot of developers, especially when the resources are an issue.

And it's being used in things like trackers that have to run for like, you know, tracking animals. Mm-Hmm. Um, it's being used in firmware on things like laptops.

Um, so all of Google Chromebooks right now this year, um, have Zephyr running on the firmware when they're powered off. Oh, Interesting. So that 'cause it sips the battery as opposed to Linux, which is just to be more resources just to keep it not awake.

Um, over in the Zephyr booth, you'll actually see a framework laptop, which is a doit, well, it's a build yourself tip of laptop, or you can buy a built and everything's full in play. And they're using Zephyr too. Um, we're also in Oticon hearing aids.

Mm-Hmm. And it's also in wind turbines. So there's a lot of places people don't, It's a lot of places, but never think about.

And, you know, this is where people are running software and they wanna have good security. Yeah. And in some cases they're wanted having safety.

So let's talk talking about both. Yeah. First security.

Mm-Hmm. One of the challenges we've had ages ago, I think we've done some things to help with this, is how do we, how do we continue to update Yeah. Added systems at, at, And So for many, many years it was sort of the usability life, uh, uh, uh, span of a piece of hardware was it's no problem.

We'll, we will obsolete it, replace it hard to do in a satellite or some other environment like That. And so, um, there's a variety of, um, for other projects we collaborate with that do secure boot and updates. And so those technologies are out there and we're certainly working with leveraging them.

Um, one of the other things that's important is don't do an update unless you really need to. Oh, okay. Okay.

And so one of the things that we're doing in the Zephyr project is we can automatically on every build, generate out a Build SBOM. And so if you turn on an option, you config, you gain this build bu so you, you're getting this bill dust bonus, so you know exactly which c files made into which a and which a were linked together to make your alpha image. Okay.

Interesting. And so you have that high precision, which eliminates a whole class of false positives on the security side. So, you know, if there, if the file with has a bug, the vulnerability is, isn't present, you don't have to do anything.

And being able to cut that out and just say, okay, yeah, no, we're fine and be authoritative. Mm-Hmm. Is a good thing.

So zephyr's been embracing all the best security practices it can find. It actually got the gold badge for, um, it's open SS F'S badging program. It got it way back in 2019, I believe.

Okay. Um, it's actually one of the few open source projects is a CVE numbering authority with Mitre. And so, you know, and, and we have its own piece or team for incident responses and assigning vulnerabilities.

So the project is trying to follow best practices as we know them. And it's played off for the project in the sense it's continued to grow. The, the big challenge right now is going after safety though.

Okay. Tell us some more about that. It's okay.

Kind of popularized now more people are thinking about it because of ai, but there's more aspects to it than, Yeah. So, so to do safety, you have to look at a system, you have to look at a full system of components. And that is, part of that is softwares, but it's also what software, what hardware is it running on, what services may be pulling down.

And as we start to bring AI and modeling and things like that in, you know, what are the implications of your data sets and your training methodologies? So with Zephyr, what we're looking at is, um, getting it so that we can take the kernel of it through formal safety certification. So if we used in 60, like the certification we're going after is 61 5 0 8.

Um, we've been working with the certification body already. We're in our phase one assessments, and we'll be working on our code base through this next year. And we're hoping to get 61 5 0 8 and 26 26 2 A LD, um, next year.

And so this is what we're actively working on in the project and trying to figure out a methodology where we can do this in the open. Because right now a lot of the safety stuff happens behind closed doors and under NDAs because of the liability reasons. But because we don't have any, because we're open source project Yeah.

We're more transparent. We can be more transparent. We can try to show people how to do it.

And so if we can do a bit of it, we are hoping that other people will help, help us build on it, what we do. And so That's, we can get more people to do it once they understand it's Not exactly, or, or understand it and then see the process. In fact, Zephyr is one of the projects that's actually trying to do this.

There are two others I work with that are trying to do this, one of which is a Linux kernel. Okay. And there's a project called elisa, enabling Linux and Safety Critical Applications.

Okay. Okay. That's a, that's the acronym not To be mistaken by the Macintosh Elisa.

No, Predecessor. Elisa. Okay.

Okay. Elisa. And so that group is a group of people that have come together to collaborate on how we can, um, surface up requirements, do the analysis of systems.

And so we've got some working groups in automotive and medical devices and, and aerospace that are meeting and trying to look at the problem and then trying to figure out, okay, how do we configure Linux? How do we assemble things together? What are reference systems?

And so they're trying to tackle the, so a lot of competitors are trying to collaborate on making things a lot stronger. Okay. Uh, they all need to solve the safety problem.

They all need to solve the same problem. They're all working with Linux and they're trying to figure out how do we take Linux into a, you know, in spaces in places we need safety. Mm-Hmm.

And then the third one that's trying to do this beyond is Zen, which is hypervisor. Okay. And so it's been around for 20 years plus.

Yeah. Familiar. And so the maintainers in that community are keen on doing this.

And so they're working on, you know, updating some of the code base to be aligning with RA coating standards. And then they're also looking at how do they start surfacing out the requirements so that we could have the requirements, the code to the tests, to the evidence to show that the requirements are satisfied. Mm-Hmm.

And then eventually, if all these projects are able to surface their requirements, well then when you create a system, uh, you've got, okay, I've gotta use this sensor. I've got on this operating system, I've got this property I need to satisfy from these things. When we have to plug these open source pieces in, we can hopefully connect our requirements.

So you can start to see, oh, if I use, I use this sensor, et cetera, here are all the tests I need to run. You Kinda have traceability. Right.

Exactly. And that traceability is what the analysis needs for the standards. Mm-Hmm.

And so as we work our ways towards that, having that transparency is going to be key. And one of the other projects I work with is to try to improve that transparency. Okay.

Is a project called SPDX or software package data exchange, except it used to be called software. Now it's called system Packaged Data exchange. Oh, interesting.

Okay. Because it's more the software, right. We have, we're having to track metadata.

It's important to know which hardware your software is running on, because there could be a bug in the hardware. There could be like, um, do you remember when we had, um, specter and Meltdown? I do.

Yes, I do. And so it was the, it was, the problem was in the prediction side of the chips architecture. Mm-Hmm.

And they had to work around it in the kernel and firmware to make sure that it wouldn't, couldn't be exposed. And so you had to know exactly which firmware you had. You had to know exactly which hardware you had a running on, which, what chips are you running on.

And then you also have to know the versions of the kernel to know that you had a safe configuration or a secure configuration. That's A lot of parameters. You don't always don't Normally have to.

And normally right now application right now, now, right now, they're all scattered in, you know, different things. Like some of the, we're starting to get the SBO stuff working. Okay.

Mm-Hmm. But, you know, the stuff for the, how the track, how we're tracking the firmware and the hardware together and things like that is still happening very manually inside organizations. And then they're sort of trying to layer the software on.

So this is why we need to take it up to the system level so we can actually get all this stuff pulled together in a common way so we can reason about it. Find it system. Right.

Exactly. It's not an isolated component that strictly operates other, it wouldn't do anything. Yeah.

And like, you know, one of the things you find in a supply chain is my product is your component. Mm. And vice versa.

Me, a good point. Products for me to products and Yeah. And, and so subsystem figuring out, making sure that things are modular and things can come, like, so those who have the expertise can put the right level of metadata in automatically.

We'll build trust. Mm-Hmm. And then if people can build on trusted components and start to assemble their systems, we can do the right level analysis to keep us all safe.

And it's a way, but figuring out how we can summarize all these different elements and have a common framework is one of the challenges that the SPDX community has been working on for the last three years. And so, um, we've been working with, um, people from the, uh, the, the, we've had collaboration between OMG and the CIS group at OMG, along with the SPDX community from the last first release of our spec to try to come up with an internal model so that we can actually go out in multiple formats and quite frankly, be in databases so that everyone can query, like we can eventually query this up. And we've added profiles such that if you care about hardware, these are the additional fields you need.

If you care about software, these are the additional fields you need. Right. So we're trying to make it more approachable and efficient, you know, what systems What, and I'm not an embedded systems first myself book.

Okay. Kind, kind of learning about it and reading about it. Well, the things I remember reading is that, um, for example, with things that go into space Mm-Hmm.

We're much more likely to use something that's been around for a long time. Well proven. We know the failure rate is really low.

Like maybe power CPC chicks for CPUs, et cetera. So we have a lot of legacy. Yep.

Well proven technology. Um, and then we also of course have, you know, what about the AI chips and new things that are coming down the pike and how do we secure those? Do you, how far back are you able to step into existing technology that's been out there for a while?

Or do we just pretty much let that be what it is and kind of work on the next generation? I Think, I think it's, it's working on the next generation, like the European Space Agency has already said they're gonna go with risk five. Okay.

Okay. Some of those legacy things are not there with risk five. Okay.

Interesting. Yeah. Right.

They're really make an leap. And So, you know, um, they're, you know, they, they were in MIPS for a long time and then number, we went to risk five. So they're basically building up on that infrastructure.

And so the question then is, okay, what software are they gonna be running on these things? And then how do we take and prove it? It's gonna be safe for use and things like that.

Because, you know, the more you know the consequences of getting things wrongs out there, are you throwing several million? It's, it's a financial consequence. Usually.

It's really tough to bring it back. It's tough to, well it's tough to update it sometimes too. 'cause it's a rather thin pipe to get all the way up there.

Mm-Hmm. And very slow. Good point.

But, you know, some of the prototyping work that NASA's doing it is like, um, you know, curiosity and the mo the, the Mars rovers, you know? Mm-Hmm. It has been running on the planet bars.

Right. And it's been, you know, in various places in the space, in the space station and so forth. So open source is sort of there, but it's not in the critical spaces right now.

And I think getting it so that we build confidence that it can go in those places is a function of being able to do the analysis that they expect of a system level. Interesting. Does, does with embedded systems.

Now, there may not be millions of satellites that use the same chips, but like an I OT kind of the other end of the spectrum, you know, it can be literal, it can be millions of devices have that embedded system in it. What kind of complexity does that bring to how you think about also Security software? There's delivery.

Yeah. So there, there's, there's businesses and lots of startups forming about managing, you know, um, Goliath, which is one of the members of the, of the Zephyr project. And there's a few other members that are actually quite good about, um, talking about and having solutions here where they basically, you know, help you manage from a control point of, and you know, what, what, what versions of things are, which versions of the chips and how you update them and be consistent.

So it's a business that's been formed and there's a lot of people that are interested in starting to go out that, you know, you have a whole fleet of trackers, you know, you're tracing, you know, you're tracking through your food supply chains or you're tracking through your vaccines. You make sure they don't go about temperature range as they're shipping out, so it doesn't invalidate them. These sorts of trackers are already out there and are being managed.

And more and more of those are showing up. We live in a world of sensors where we realize it. We've Got lots of sensors out there, they're wonderful, uh, you know, but we need to be efficient with them.

And, you know, there's ones that are basically powered by solar Mm-Hmm. Solar powered trackers. And, you know, you see things like, you know, energy converters even see some things.

I'm starting to see some things about passive powering because it's tipping that low that they can do the passive powering from vibrations and other things Mm-Hmm. To actually power things. Interesting.

So there's, there's lots of really cool applications in the, in the deep embedded space. Yes. That's fun.

Yeah, it's cool. What's, What's the next problem you'd like to work on? Oh, I think the problem I've got of trying to get open source to, uh, work all this, uh, safety critical infrastructure, I think is a good problem.

Five. I think that one's good enough. Okay.

And, and, you know, I've worked with the SPDX community to make sure we can start to talk about hardware more efficiently. Talked about virtual twins or digital twins more efficiently. Mm-Hmm.

Um, and be able to pair up the hardware with the software and like, you know, you, and you know, quite frankly, hardware is composed of other hardwares composed of other hardware. Right. And, you know, can we describe a hardware supply chain with, with, uh, a common metadata format such that we can reason from, you know, this piece of software to this chip on this board, on this system and make the connection.

You know, I think being able to, when you need to make that connection, you know, when someone's, you know, the f FDA is gonna be looking for things like this already we've seen the things and there critical infrastructure, like the energy sector is looking for this sort of stuff. So it's a good challenge. It'll keep me busy.

It's a big challenge. Do we need to redefine SBO m system bill of materials? Yes.

So we covered any instance, whether it's it's only software, Whatever combination. No, I, what we need to basically re-look at the whole problem as a system problem, not just as a software problem. Now the software bill of materials is shining a light on a, a part of this system that was opaque before and there's work out there already for hardware building materials.

People know how to do that. The supply chain's been doing that for years. You've doing that a long time.

Yeah, right. But let's make sure we can figure a way that there's a common language so they can all talk to each other to know exactly which piece of software is deployed on which piece of hardware. And that people are someone, someone like, you know, an organization that's tracking all these devices is keeping in track about all to make sure that, oh, this, this set of the devices has this piece of firmware on it and oh, and it's gonna need this security update.

And, but oh, these ones here are a more recent ship, therefore they're fine. You know, being able to get to that level of precision will help everyone. Very good.

Well, okay, I know you've got a panel to go to, but folks wanna get involved. This is an interesting area. Okay.

How do they get ahold of you? Or get ahold of Yeah. Look at the projects.

They might wanna be, Um, they might wanna take a look at the ELISA project, if they care about Linux. Um, they might wanna take a look at the Zephyr project if they're having that resource constrained considerations. And then, um, if they're interested in making sure we represent hardware properly, if they're interested in making sure we reference services and some of the operational stuff properly, um, the SPDX project is always welcoming the community to come in and join us.

And if you look up SPDX on our GitHub repo, uh, all our meetings are listed, all our mail listserv are there and you can just basically self sign up. So we try to make a very low, easy barrier to entry. Very Good.

Sounds like you need help. You're, you're welcome. Help, well we always Oh, very much welcoming.

Help any of these projects Very much welcome. Help. Very good expertise.

Oh, it's nice to talk with you, fa good luck with your panel. Thank you very much. It's all about, um, we've been pre we've been moving the, um, preempt RT patches into the upstream and this has been work that's been happening for about eight years and it's almost finally done.

So we're gonna be talking a little bit about that today. Some Of those problems stick around for a while. Well, the hard problems take a long time to solve properly and we have to solve them properly.

Yeah. Otherwise, the solution doesn't stick Well. Why solve 'em multiple times.

Right. That's what was happening. Do it right the first time.

Well, thanks again. Good luck with the panel. Thank you very much.

Enjoyed talking you with The Linux Foundation. Another great conversation. You know, it's, to me, it's fascinating the, the melding of hardware and software and thinking about embedded systems, even though that's not my special thing.

So many cool things going on there. So it'll evolve many, many projects you can contribute to. Thanks Jake.

Thank you. We'll be back with more interviews and um, Margaret, good stuff from the open, uh, open Software Summit. Oh yes, there we go.

I don't know if I, I can get that right every time, but, And we're also running embedded open source summit here in parallel. Oh you are? There's lots of embedded People.

There are lots of parallel. Yeah. Yeah.

So you see they're two different T-shirts, even though behind you, Do you ever get the T-shirt with different ones on both sides? 'cause you're playing multiple roles, right? Yeah.

Well I do play multiple roles. I enjoy it. So That's great.

Thank you so much, you again. You bet. Okay.

Alright.