Philipp Ahmann, Bosch | OSS North America 2023

This is Techstrong tv. Welcome back to the Open source summit. We're back here in beautiful downtown Vancouver with Philip Ahman, who's from Bosch.

And we're talking about embedded Lennox in safety applications, otherwise known as Alisa. Right? Right.

Yeah. So we've been putting embedded Lennox in things for years. So what exactly is different here?

Why do we need this project? What makes safety applications somewhat different? Yeah, safety applications are crucial.

Maybe because you need to talk about the threat to human life and which you would like to prevent this, you just need to manage the risk, which could happen. And if you take a normal embedded system and if you maybe talk about security, you just want to prevent the data get lost, that someone can intrude into the system by here. It's about that the system need to be safe in a way that it cannot harm others.

So you need to take a lot of measures. And for this, they're best practice in place typically with standards, international standards like the ISO 2 6 2 62 or an ISC 6 15 0 8, which are traditional standards. And they are developed by companies for as a best practice of their way of going through processes.

But we know that open source is not developed in the same way like you would do maybe a company based product. And so this is something new, especially when it comes to Linux because Linux brings a very, very large code base. And typically the safety applications are trimmed down to the minimal thing which you could do.

Like we don't have configurations normally in a product. This is something where we come from embedded small devices, but now use cases get more complexity grows. And by this we are on the way, like how can we get safety into an operating system as rich as Linux for us?

And that other can benefit from it when building more complex devices going into driving this or uh, robotic use cases, aerospace and so on. All Right. So we can't have a scenario where if I'm driving down the road, the operating system crashes and suddenly I crash because I, the operating system crashed.

So yeah, what does it take to actually achieve that level of robustness inside that? Because everything eventually breaks somewhere. So Yes, exactly.

And um, so this is why you really do extensive testing. You write on requirements. You basically make sure you do a lot of risk assessment risk and all of this.

And then when you start to talk about risk, the things just evaluate naturally. So you see all these practices which are written down in the standards and you may want to how they do not apply one-on-one. But if you just start in your development and you start talking with another person and I say, oh, okay, when I explain something to you, let me write down a design.

So then I start writing a design and when another one comes and say, I just see this design in this perspective, no one sees another perspective, then you add requirements to the whole thing and say Okay, we have now base, we derive things on top and we go forward until we create such a system. And this makes a large chance also opportunity to make it more robust because you ask what is in there? Let me check, is my testing on a requirements level on design level, implementation level all satisfied and can I achieve robustness stability over a long lifetime?

And basically this is something which you already have today. A lot of the infotainment system navigation devices, there's Linux space underneath. And already 10 years back we had the requirements that the system must not crash and it had to start in two seconds.

It had to fulfill a lot of stability and it need to behave today as in 10 years because the car for example, runs for 10 years where you don't throw away your mob like your mobile phone and all these things which we have put into the Linux ecosystem for example, we also experience in our devices, if you go for an Android phone 10 years back, you had to switch it on, on and off maybe once a day or every other day. And uh, my mother recently asked my, when should I switch on and off my phone? Uh, is there a time where I should bring it up?

And said, well, when you feel that it doesn't behave us before, and said, oh, I wonder if you switch it on. I have no clue. So I looked into it and it could go to the system saying like, was something like 6,500 hours without a reboot for the phone, right?

Like that's a massive amount of time. Howand Android system which has Linux kerlan needs, runs stable and all these service and, but it is not enough for safety, right? It's just that you need to add much more checks, much more robustness, freedom from interference and these kind of things and a lot of hours working on it.

And also testing. I would've said to your mother, if you feel like cursing at it, that's the time to reboot it. Right.

Are there different use cases for embedded Lennox in all these different vertical industries that we discussed? Is there a common thread in their requirements or is the medical device and the car and the rocket fundamentally different? Yeah, So there, there are differences of course if you talk about sensors, actuators, but they also share a lot of things.

It's the scheduling. Uh, some have maybe stronger realtime demand than other use cases where address one use case which may not even need time patches in there and still has a safety responsibility. But it's just that the safety doesn't need to come within a milliseconds or microsecond but a little longer.

And, but what you can see for a lot of these use cases is it's often that something goes into a memory and you compare against another memory region and you have to fulfill this in a given amount of time. There's a little bit of oversimplification, but it tells you you need to look into how Cassius are handled, how the memory works, the scheduling guaranteed and is there something which could cause interference. An example would be, uh, if you think about the graphics GPU and if there's a workload on GPU and an embedded device, it may heat up if you play a game on your phone, it's just getting hot and it may go then that the CPU uh, frequency scales down to get lower temperatures.

It's not giving full speed anymore. But maybe you had an assumption that I have a guaranteed frequency and that's also which my workload is scheduled on. So the time may sense.

And then you need some kind of watch doc scheduling observation of the system and these kind of things you need to think about. And this is similar from me, from medical devices, aerospace, wherever you go industry, you need to have guaranteed timings for your workloads and you need to make sure that the memory is consistent. Because if you cannot trust the memory, you don't know what's going on.

So will people take um, an instance of Linux that an embedded straight from you guys or are they gonna take the core of it and then customize it and extend it for their use case per se, but then at least they're reducing the total cost of building something because there are those common functionalities? Exactly. And basically they need to do the same thing as we do.

And we also just, we don't provide a safe Linux. This is support is enabling Linux and safety applications. So we look what are tools, what do we have to improve in documentation?

We, we recently figured out that tracing workloads in your system is a crucial element. And for this, uh, there was an upstream in six to three kernel how to do workload tracing with C scope and F trace or so that you just see, okay, here are elements in there, this is subsystem which are important. And if you build your system, you can use this tracing to understand how your system operates.

And this can also benefit not only security, uh, safety, it can also a benefit for security or other critical workloads which Iran, so this is something which we bring in and where you then get a benefit later on. Are you looking at security as part of this Exercise? Yeah, we, we always touch point with security.

We don't put a focus on this because there's so many security rated projects out there. But uh, for example, looked in trusted execution environment and hardware acceleration for security because security mechanisms which reduce the rights which you have in the system will also benefit in a safety system because you need to make sure that your workload gets isolated and made a difference, which you have if you say my system is secure, you may look into your little small bubble of it and say, my bubble is secure, I protect what's inside. It's much treasure.

Mm-hmm. But this is something which doesn't hold true for safety because it could be another system which just get impacting, not directly but indirectly as I mentioned like with CPU load or temperature reason or uh, a shared resource which is not really going into the worker, but you share the same interface and this is where differences come in. But security mechanisms definitely help also secure boot to make sure that your system is robust and in a defined state.

And here we use a lot of these security mechanisms, well name spaces C groups will also be something which can be utilized in uh, safety rate system. And we have actually a working group on this which concentrates on Linux features which may benefit in a safety argumentation. And they also use mentorship programs from the Linux Foundation to have a mentee, which then also maybe takes more, little bit more the security side and see how artists security apply to safety work.

Right. What level of skill does someone have to have to build these types of applications? Cuz it seems like, you know, unlike say working in the cloud where I've got some, you know, infinite amount of capacity and I can be a little bit sloppy feels like in this environment I gotta be really precise.

Yeah. So it's even going that far in the traditional field of uh, safety products, you will not have dynamic memory allocation so you know before which memory amount you will need. And you do aesthetically assign this during the boot up so that there's no swap needed, no dynamic memory allocation at all.

So you need to have a very good system understanding on how the hardware works. We have hardware experts from an invalid field, you need to have Linux understanding or operating system understanding how to control us work, how to, how do my system overall operate. And then this will not help you if you don't get all these safety aspects.

And as well they will ask what the risk, what could go wrong, where are harms, what are the interfaces? And interface does not only mean the interface between two functions or so. It also means how a human interacts with the system so that you have a much wider analysis, analyze fall trees, what could go wrong, what could be a fault in the system.

So get all these kind of things in and I'm, I see that typically you will not have one person who could build it. So you need very much size of a team to interact, to cooperate in this. And um, I see this quite often that people approach the laser project can say, so you will build a safe Linux.

I want to build a safety related product, I want to use Linux for it. And then you figure out the people don't bring a strong Linux background, they don't bring a strong safety background and they believe they can get something and we decide that we are not delivering a safety Linux just enabling Linux and safety application. This also means even if a company would provide you with a safe Linux and they are like, uh, sus Red Hat canonical, they all work on this past ours functional safety.

Linux nice to meet you even if you buy it from them. They need to have a sufficient understanding of the system of what is in there. But uh, but here opensource is a great enabler because we have such a huge amount of experts in there and you have the chance to learn it.

When you take traditional safety operating system, they are often delivered as binaries. So you have no chance to look into it and you need to, you just get the manual and then you need to trust on the behavior which you experience. And here you get the chance to really see into and it's related project like Sapphire or Xen also provides, they have an open source coat base, they have a strong community with people contributing to it with various expertise.

And this gives a large strengths, which you may not see in traditional Commercial models. You may, you may get surprised in a space where no surprises are kind of crucial. Yeah, right.

That's, but what also, when you see no players, new players coming in, suddenly they may talk and say, imagine you want to update something and you want to update your break. And that's, I say I do not want to update my break in the, in the system and if I'm in a car, I want to have a functional break it at the time when it's released it should be safe. It has to be safe because it's the human life threat.

If it, my brake doesn't work. But of course if you see connected devices, IOT and some the area we're working, suddenly everything gets connected and you may want to get an update or a new function in your system is, it could mean that it's not an update for the brake, but it could be a new driving assistance function because the sensors are in there and then it's something where somebody Linux gets a strong benefit and what you have, because there you have a chance of having open source, having update ability. But this is also a challenge which just goes in, Does someone or some entity need to certify these implementations or, and is that in each vertical industry or is there some way to approach that?

So Uh, you need to, you need to certify whenever you build this and you go with certification authorities, you take a choice of the safety integrity standard, which you would like to apply it. So when the ISO 2 62 62 is this standard for the automotive, you can also argue that you take an IC 6 50 0 8 standard, which is just the upper like say the mother standard of it. You need to convince the authorities that you do the proper thing and this standards help you.

It's basically showing you state of the art development. And this is very complex because they say, oh do you bring a safety culture? Do you have follow the process and so on.

This is all in there. This makes it complicated for using Linux in there but also if you just do your component development on top, but you have more control if you are in your company and build it in there. And this may also give a benefit for these strong people like just mentioned like canonical Suzy and redhead.

They bring a strong background in processes. They have this, they have industry support or they the indu industrial branches of Linux. And by this they can much easier and Apple than maybe a new player.

A new kid on the blog which just starts want to develop it. Then I would recommend just better go a bit smaller. Don't try too much on it.

Right. Don't boil the ocean. Yeah.

So what exactly do you guys need from the people who are watching this? Do you need more contributors? Do you need more companies to participate?

What are you looking For? What we, we were really looking for were an industrial area because we currently see that due to the RT patches going into Linux, there are more industrial automation people who ask for Linux and they also have safety standards involved others one which we then we addressed. So it would be very nice to get some more uh, visibility also in this industry because it will help us, it gives us a different track and it's what we see.

We got a good safety background, but bringing this on the ground, so we, we started with a Q o emulation, so it's a virtual machine because this was easy to scale, to scale to share, but to bring it to hardware also requires a lot of embedded engineers, kernel engineers. And we would like to love to get more kernel embedded developers involved where we can help and train them also on what is important for safety. Because we did a lot of these and all of those parts, we've got a good understanding.

But to show it more in a, here is the Singapore computer and you can just see the workload in there and we go out having a booth like others on a, on a conference. We were hesitating because we wanted to show more. And for this, my contribution is good.

Also automotive industry, um, driving force, no going into driver assistance systems. Right. And we by intention decided for another use case in the beginning we were using the warning signs, which you know, from your car to gear indicator, the check engine orzech oil part, they are good because it's easy to explain less sensors, less actuators.

But that's not the fancy thing that people want to see, right? You want to see driving assistance and that's something you can sell with marketing, it's you can go out and for this it would laugh to see if there is an automotive OEM maybe who comes in and say I have my use case and I'm willing to openly share ideas, I will do my development maybe in-house, but I share with your concept and maybe even something which is 10 years old because we have driving assistant, which is 10, 15 years old and they just say that's how we did our analysis. That's how we did the work.

And then combined with embedded engineers, we, because say let's try to bring this on Linux and argue why this can be sufficiently be safe in this use cases. So here community support, especially from embedded developers, from engineers, they would be helpful. So everybody wants to be with the cool kids and they kind of focus on Kubernetes or the mainstream Lennox project.

So how do we make safety sexy and get people like say, hey, this is the place to be. I guess where Where it comes in a little bit is that people talk about the software defined vehicle, uh, and many different, some call it like the smartphone on wheels, others say it's subscription for your seat heating. But uh, here suddenly also cloud technology comes into picture where you go for digital twins and this is something but I see as a benefit, it's not as sexy as the other topic that's, it's like you don't, you cannot uh, run fast and fail and just start over because you do need to do it right.

But you will get a very good understanding on the software engineering process. You will write better software by it. You will get a good understanding on things and let you rethink how you do code.

And I guess it's a benefit to all industry and it's can even increase kernel stability, reliability, robustness. And you will, I guess it's more the field who really would like to touch something in hardware where they say I would like to learn about and there Linux is much nicer. Also other open source projects as many like exact far because you can try things out.

You can see when you do something or suddenly it starts blinking. You can push a button and something happens and these are other things which are hardly to grab in the virtual space. You see something, it's rendering there, but I get to know this, we were running with Q or we hads on a pc.

If I go somewhere demonstrated in the industry, they say yes, nice, but how is this a product? We will make a device which runs in millions of cars. And what we established from the last generation, the Bosch infotainment, I can say the Linux system, which we made up, ended up in 30 million devices on the road for navigation.

And this is something which you can receive, you can see and point to something. If you go to the cloud, it's like here's this web store, I did something around it or here's this other part. But if you build these embedded devices, you can say, look like Chuck Warr was presenting yesterday on the Boeing part, how they're using it.

Say see, whenever I enter a Boeing plane, I know something in this plane is there for me, my side. And that's something which is really nice to see. So that is more impressive I see.

Than sometimes the local. Yeah. All right folks, if you wanna save lives, call Philip cuz he needs some help and that's a really cool job at the end of the day.

Philip, thanks for coming by. Yeah, Thanks a lot. All right.

And we'll be back in a minute.