Kate Stewart, The Linux Foundation | Open Source Summit Europe 2022

This is texturing TV. Hey folks, we're back at the open sophomore Summit in Dublin. It's lovely day.

We're here with Kate Stewart and we're talking about. Real-time operating systems that will run on the smallest devices. You can imagine it's called project Zephyr.

Okay, welcome to the show. Thank you very much. Glad to be here.

What exactly is the challenge that project Zephyr is trying to address because we have had Linux for all these years. So why do we need a real-time operating system project as an extension or complement to that? Well Luke's kernel doesn't get much smaller than about two Megs these days and that's really trying hard to get it down.

There's a lot of information and a lot of options in new length kernel and when you're really tight on resources you can eat to be smaller like in the tens of case tens to hundreds of case depending on what you break in. So Zephyr has been started from the you know, and designed from the ground up to be in that range. And everything is sort of config in.

So but then everything is also tested so it makes you give you a really quick start to actually getting your application working in a new space. And it's got from it's very familiar for Linux developers and metted Linux developers because it's using K config, which is in the kernel as well. And it's also using device tree, which is also in the kernel as well.

So you have the ways of specifying the devices the kernel developers are familiar with the code structuring is fairly intuitive for most developers. So we have a lot of developers that have been very active and embedded Linux for a long time that now's work with us in Zephyr. We hear a lot about Edge Computing.

And the last time I lost count there's like five different flavors of edge Computing depending on what your definition is with the network all the way down to the devices. You're talking about. How far up will Zephyr scale and when when do I kind of think about something else?

I'd say Zephyr will scale up to you know, it definitely scales into the edge if they want to we see things like people doing gateways and using Zephyr on one core for handling things that the Bluetooth traffic and using another core running Ubuntu or something like that. So you get these mixed critic mixed systems with Zephyr in on some of the cores and some of the functionality you also get it's got like a 64-bit core there. Okay, and so like it has, you know, it has both the arm and the risk five 64-bit cores fully supported.

So this you know, it can be used in these places. But where it's designed for is so that you can configure it down when you need to. Minus turbo this morning describes.

Some of the embedded system stuff is kind of like the Wild Wild West it's starting to get tamed from his perspective and we're starting to see more standardization back in the day. You had multiple real-time operating systems running on all kinds of weird Hardware. So is this part of that whole standardization effort and it's part of building a building consensus so we can apply best practices.

That like so like the last kernel and some of the other projects have evolved really good practices for high quality code. And we wanted to place in the embedded space to pick them up. And so when we serve started this effort project that was one of the goals.

We wanted to be first class and Security First Class and be able to go after safety certification. with this OS and to a large extent Zephyrs actually one of the few open source projects that is a cve numbering Authority or CNA with miter. Formally listed there.

We manage our own vulnerabilities. We can handle bulk vulnerability reports. And we have a p13 that's active in Zephyr.

It's helping to support we also have all our policies publicly documented. the open source Sorry the open source security Foundation open ssf. They picked up the best practices badge and they've got the best practices badge and there's like different levels.

There's passing silver and gold. Zephyr has been gold for several years and that's publicly documenting all the processes. So whenever we found a good security process or Best practice we tried to adopt it into Zephyr.

and that served us well, and we've had Fairly clean code base and at this point the Linus kernels about 9 to 10 commits an hour. into the code base zuffers at two commits an hour in the code base. Wow.

And so as a result, you've got a lot of momentum a lot of new features a lot of new boards. There's over 400 boards now in Zephyr Upstream. So if you're working in those resource constraint spaces, you've got a pretty good chance of having a starting point.

Start from that's already Upstream. Whoo, are they contributors to this project than the maintainers? Where did they come from?

Are they coming from the vendor Community or they independent folks are they all over the place? Like like the Linux kernel? We have corporate contributed like we have corporations that have business strategies around it.

And we also have obvious in Zephyr. I'd say probably about 10% are people doing hobbyists because I enjoy working in the code base or they have things that they want to keep going. And you know, some of them are doing maintainer type of roles as well as hobbyist.

It's a good mix now. It's very diverse. We've got silicon vendors we've got and product makers.

You know, it goes from like one of our Platinum members in the project is oticon. Which is we're still hearing aids. And you want your battery to last a long time.

You're hearing aid, right? And so they've got some of the best tinnitus support out there from what we know and they're doing it all now in Zephyr. All right, so you're telling me that the guys over here behind the camera will be able to talk to me in my mic through this real time operating.

Yeah. Yeah and using Bluetooth without having all the wires over on everything. Yeah, so and the other thing like Zephyr we've been doing a lot of work with like the Bluetooth sink for the connectivity and so A lot of the new standards and they're coming out of Bluetooth Sig they get developed in prototyped on Zephyr before they come public.

They have to be prototyped on some and then as soon as they're public they're in Zephyr Street. So we're right on the edge on that some of that stuff and there's more connectivity. We want to have in there too.

What should developers be thinking about in terms of those best practices for building these types of applications? It can be really challenging to build something that small so yeah what you know, where should they be thinking about the gotches or some things that they you know pieces of advice that you might want to give so I think the pieces of advice is you know Try to keep it as small as possible and you're configs understand exactly what you're building in and make sure that's Justified. Because it's easy to just serve a clone copy paste.

But if you're really trying to get down this small thing, it's very small footprints. You know scrutinize those configs and make sure that everything that you're configuring in is actually needed. other things that we should be looking at are You know when you're compiling it building it again.

Look at your options be very precise about your options. We've got a lot of ways of you know doing this type of work and optimizing or not optimizing depending on the compiler options again, look at focusing on what you're optimizing for. But is it time or is it space?

Because sometimes there's trade-offs. I'm not accusing anybody of anything but it seems like historically there's a been a lot of large code a lot of sloppy code that people had to write because they had a lot extra Hardware to throw around. So are we going to see a new generation of folks who are writing code in a much more precise fashion almost a throwback to the old days when we had a little bit of CPU and it was all about.

Yeah. How efficient is your code? Well, let's see I think There's a lot of modules and components out there for the fast prototyping and doing things like that.

But then once it becomes like, you know fractions of sense count and they don't want to put a little bit of RAM on you can configure it down and you can you can prototype quickly and then you can optimize it efficiently over time. And I think that's the sweet spot for the industry. Because you know, we're all getting really concerned about power and energy.

And you know, you want your devices to last a long time. And you know people putting software upgrades in for firmware, you know for vulnerability fixes things like that. You want to make sure we still keep within a power budget.

You still have the ability to do these types of tasks. We heard a lot this week so far about devsecops and s-bombs and all the things that are required to build a secure application. And these Edge applications seems to me, you know, I don't want somebody inadvertently puffing stuff into my ear to tell me something because they hacked the microphone from these guys.

Um, so how do I kind of think about that as I'm building these applications and work that into my whole workflow? well here The ability to just say West SPD West is the build tool in Zephyr. Okay, and it's based on cmake.

okay, so the same age infrastructure is there and you basically do a priming step of one line command and then you do your build and then out of the butt end of it. You get three s bombs two sources bombs one for your app one for this after sources and it build us bomb that links all of the pieces together. So, you know exactly which dotto's.

And they correlate back to exactly which source files have made it into a DOT a and those dot is which makes it into your Alpha images. and so the actual image that you'll be putting onto your Blast about you know until you're flash device you have very precise information about which files are there. So if someone said, oh there's a vulnerability enough net and you go well in that version of else, you know in the in the LTS one for Zephyr.

There was fnet component. Well the files that were part of that vulnerability in that component. If they're not in Zephyr, which they weren't actually they could never be in that image.

So even though you could say the component might be there in part. You have to be at the file level to be very precise to know especially when it when Size Matters like this, you know, you're not pulling a thousand packages in from the internet. You're figuring out exactly what you're putting on your device and putting it down there.

And that Precision is key. We've talked a lot about software. But do you think that this will drive a level of innovation on the hardware side because there will be standard interfaces and those guys won't always have to go build the entire stack right?

There's a lot of innovation happening on the hardware side right now. We're seeing a lot of really cool things happening in the fpga front and the accelerated front. And we're seeing people using Zephyr to enable this functionality.

And so, you know as we did and we're seeing simulators being used to you know, do all the prototyping and so forth and then manufacturing exactly and the risk five space in particular exactly what they need what accelerates they need and then having the enablement right there in the hardware like the software. Coincident with the new hardware so these types of stacks and this type of capability. Is going to be changing things a lot, especially for power.

And you may have modules that have say an arm processor and an fpga over there somewhere and and that whole thing is a distributed. Microcosm of a solution and you're using all these different components for different optimizations. Does that put more onus on the developer to understand what the hardware can and can't do or can that all be hidden in a compiler somewhere?

Well Zephyr. Support variety of different cores, so when you're starting to talk multifunctionality and all those things you generally covered and then you say okay. I want this functionally.

I'm working on this core which is you know, we've seen soc's develop now that like have you know, a risk five on One Core in the scene in an arm around the other core and the society I think was a Vegas stuff from NXT. And you know, we see people doing this heterogeneous type of computing based on what they're trying to do what what's being accomplished and the fact that in the same code base you can choose which pieces are there and you could be explicit about what's going there is is making these things possible for effective communication. We've seen sophomore kind of evolving in the subscription model.

Well that continue all the way out to these very small endpoints because I'll be able to say to people well now that you're connected to the internet I can deliver streaming updates to you no matter how small that device and it might lead to the personalization and democratization of all kinds of stuff. It's worth the vulnerability management is I think we're seriously services for people to keep things up to tip and provide services for deploying and monitoring like, you know sweets of iot devices and there's a vulnerability. Okay.

I want to import that tracking all those ones. We've got some of our members that are having business models in that area. And it's needed okay, the fleet type of aspect of iot deploying this stuff making it visible.

We're seeing the processes. We've got the infrastructure to support it. So the ecosystems are heading in that direction and realistically, you know in this in this space, you're either going to update it.

You're going to throw it away if there's a vulnerability. Or make sure it can't be exploited by the environment right and you know some of the devices if they're connected Internet you may be able to update but some of them you may not. and being able to be explicit about that is important for You know being able to trust what's the environment is?

There's a lot of concern about security as it relates to anything on the edge and iot in particular are there things that we're doing at the operating system level or even the hardware level to make all these environments more secure and take the stress off other people. Yeah. So there's a whole ecosystem of Trust and Believe from the very initial boots.

So this whole trusted boot trusted update there's Hardware accelerators for enabling trusted keys and effective sharing of cryptic information. And Zephyr enables working with all of these and we partner in with some of the other projects. They're working with these capabilities like, you know, the TPM and so forth and the Intel equivalents.

so we have a lot of this stuff enabled in the code base and it's there just be taking advantage, you know to be used as needed. That's part of the whole has ever be all sorts of things integrated together and test it together. And which is it's like it's not just a kernel.

It is actually oh here are these drivers that we here's this communication stack here. Is this capability? And it's all been tested as part of our ecosystem.

It's kind of a software. Corollary of a sock in some ways where it's the stack that matches the sock and we'll go from and then and yeah, and the fact that we have a lot of regression testing going on in every build well in every commit as well, as you know, before we put releases that we have various quality criteria, we have to hit. As for you know from our release Cadence and we have a Cadence about every quarter one of the lessons we took from the Linux kernel was, you know have a predictable release Cadence.

And what go and have emerged window now ours is different than the Linux kernels because Lynch causes like a two-week merch window when we actually do a three-month wrench window and then we do a one-month stabilization that seems to work well from our community. And but then the other lesson we took was like having the lts's our long term Stables. And the legs kernel so we do a long-term stable every two years as opposed to the list called.

Does it every year? And then we support it for two years with vulnerability fixes and that but back ports for bugs and vulnerabilities effectively. What is the thing that's top of mind for the community right now?

What's the the most pressing issue of the day? What do you guys really focused on at the moment? Squarely focused.

I think there's a our path towards going to safety certifications. So we've got a functional safety manager in the project. And we are trying to work.

There's no good place. We're open for open source has been taking to functional safety. And we're trying to prove it out with Zephyr.

And so there's a strong. Culture around safety and it's been very proprietary. Very waterfall not thinking about it from like it's but there's very proprietary.

mindset and safety space in the arthouse space and not a lot of the evidence is open. And then we have open source, which is moving fast innovating quickly and so forth. And so the challenge has become for us is okay.

How can we work out a flow so that we can automatically get the right evidence generated and be able to let people use them different system so they can do the effective argumentation. Yeah, and that's that's fun. Let's say for arguments sake you had a magic wand and you could change one thing about what was going on in it to help Zephyr.

What would be that one thing you'd be like, let's fix this today. Good question. I'm having a hard time with that.

Actually there's a communication that was must be wrong long. No, it's not actually it's there's a lot of things we're doing. Right?

Like I say, we're continuing we've had a process group. We continually trying to improve our own processes in the project anytime. I find a Best practice we try to incorporate it.

So I'm looking for Best Practices to you know can to work with the community the technical steering Community is what's actually determines exactly. What goes in the code base. And they're the ones that are always on the edge of looking for a new things to go in.

And how we can actually make this a better project. You know, the thing that probably would make us happier is finding more people to do maintainership roles. Like the colonel we always have there's this can always be more maintainers in there and sharing that load of it better.

You know, the traceability aspects of QA and figuring out how that's going to play with safety or things that I care about. So more maintainers we and so we can also have faster turns so people are happier. Yeah, everyone wants more maintainers.

So for folks that do want to get involved. How do they get involved? Where do they sign up join up?

And okay well, They can just basically Point themselves at the GitHub repo clone the repo and start playing. We've got a bunch of examples on, you know, getting started guides. And for people to just start, you know exploring with boards.

There's simulators out there if they don't have Hardware in their hands, they can work with the simulators and there's examples, you know, there's like Like say this. I think re note so at micro has a Reno dashboard for Zephyr and they've got I guess about over 350 boards there. Five applications already sitting there.

Most of them all pre compiled pre-built that they could be running on a simulator in just experimenting. So there's a lot of places to experiment and then if you've got a problem Start looking around. We have a very active Discord environment.

So if you've got question, how do something there's people that are pretty helpful on the Discord to interact and then the GitHub repo. So go and play with it. You know figure out if there's a problem you want to solve with it play around see if you can solve it and if you can't ask questions, and then ideally if you solved something no one else says, please Upstream it and share with everyone else.

That's my open. Source Works. Do you think as we evolve here?

We will see the equivalent of something and maybe it already exists for all I know devops best practices for building software at the edge. Yeah, I think there are definite best practices for building software at that edge. And I think we will be starting to see we can actually do there is work with the k-3s to actually do containers one of the things that's quite interesting.

And this is coming in from the safety space is mixed criticality systems and having the ability to effectively have small containers so you can abstract. Out what's below that level and have an interface you can just see your application on top of that. So we're seeing people there one move it in this direction and we're seeing you know these packages, you know, these projects are interacting to make some of the stuff all happen, but you know Zephyrs boundary is you know, how do we make ourselves in effect of our toss?

Right? So we may have tiny containers one day. Oh, yeah the Cooper three.

Okay three s stuff. Yeah. It's already sort of working with effort.

And so yeah, you have tiny containers that have abstract layers and you know. People in the want to basically validate and spend the effort on the things that they add. And they don't want to have to do everything below all this effort all the analysis and so forth.

And so if we can get ways of building these trust boundaries. then the people can innovate on top of these trust boundaries and then apply the analysis that others have done into their over situations, which will let us use these things effectively in the future. Alright, folks.

You heard it here first, if you want to be cool. You got to be on the edge. So check out this whole Zephyr initiative.

Okay. Thanks coming back.