Why eBPF Is Redefining How Software Runs Inside the Linux Kernel

Hey guys, thanks for the throw. We're here with Bill Mulligan, who is a maintainer for S Cilium, which is a project that is run by the Cloud Native Computing Foundation, but is also based on EBPF. And EBPF has a new report out talking about, well, the level of adoption and what's driving it these days.

Bill, welcome to the show. Hey, thanks for having me. Where are we on this journey with EBPF?

I feel like, you know, the technology came out and it's been a little while and it's starting to show up in the latest versions of Linux, but it's not clear to me that everybody's deployed the latest versions of Linux just yet. So where are we on the journey? Yeah, I think we're really at an exciting point for EVPF, kind of as you said, the technology's been, uh, out for about 11 years now, but that's kind of when it first got merged to Linux Kernel.

As you know, most people probably aren't running latest kernel. They're running ones that are 2, 3, 4, maybe five years back. And so as the technology has progressed, we're finally getting to the point where most major Linux distributions and LTS releases now support a modern enough version of EBPF that we can kind of start to do fun and exciting things with that.

Right before we just really small programs limited to 4K, now we can do a million instruction programs. So I would say the fun part about EDPF is now possible in production for the vast majority of companies out there. So are we gonna see some dramatic accelerations in performance of a lot of the tools that we used, the impact?

'cause I seem to remember the promise of EBPF was that I would run my, uh, security or my networking or my observability stack deeper in the kernel and performance would improve. But to your point, a lot of those apps are still up in user space. So are we gonna see something dramatic here?

Yes, that's a little bit what this new report is diving into. So one of the promises of EVPF is that speed. And the reason why kinda that promise is coming to fruition right now is because of the flexibility that EVPF provides.

So it allows us to rewrite what's happening in the kernel to do new things. So for instance, in networking, um, a lot of the times, like what we can do is skip certain parts of the networking stack. So a perfect example, uh, of the report is CloudFlare obviously well known for building a safer, better internet.

And one thing that they have to deal with a lot is DDoS attacks. So they, uh, help mitigate a 13 terabyte per second DDoS attack with the help of eeb PF. And the way that they're able to do that is the performance benefits that they get.

And the way that they're doing it with is a subset of Eeb PF called XTP or Express Data Path. And what that allows them to do is to process the packets before it hits the Linux kernel networking stack. So rather than having to process through the whole thing, they're able to say, right as the packet is coming off the neck, making the decision, should we pass this further onto the stack or should we drop it?

And by dropping it, essentially as it's coming off the off the nic, you're able to massively reduce, uh, the amount of CPU that it takes to, is this packet okay, should we process it? Where should it go? You're just saying, no, this isn't a good packet, we can just drop it.

And so they're able to mitigate that while still being able to serve you basically all the internet traffic too. So that's what kind of Eeb PF is bringing. It's, we can rewrite parts of analytics funnel that adding this flexibility that we didn't have in before.

And by doing that, we can do smart optimizations like networking. We can process things a lot sooner without going through the networking stack. And the report goes into, well now that EBPF is available to organizations basically everywhere, what are the actual things that organizations are seeing in production to improve that?

And so one example was the cloud ferry example that I gave you, um, of mitigating a huge DDoS attack. Um, and they're able to do that thanks to the flexibility and the performance of EBPF. Hmm.

So besides CloudFlare, who else is called out in the report? What are some of the hys? Yeah, so there's tons of people called out in the report.

So for instance, um, META is using EBPF for profiling. Um, by profiling all their applications, they're able to find a one character change that helped them reduce CPU performance by, uh, 20%, because that was a really often called function. Um, another example is Datadog.

Um, they're using eeb PF for both networking through cilium and security. Um, so looking at how they're able to reduce the cycles and have more fine grain control about what they're doing it. And I, I think the use of EVPF really spans across the whole ecosystem.

It'd be hard to kind of capture all the different ones in the report, um, because there are so many different examples. The companies that we did highlight though, um, we're span a wide variety of industries. So we dove deep into, uh, meta into Netflix, into by dance into CloudFlare, and also into recruiting.

So telecommunications, consumer applications, um, uh, social media, you know, the internet infrastructure, the uses of Eeb PF are almost like endless across all these industries. Mm-hmm. Um, we're of course living in the age of ai.

Do you think that the rise of ai, whether it's on the server side or I don't know, even maybe these AI agents, um, is that gonna benefit from EBPF and how so? Absolutely. Um, I, I think if you're looking at ai, it, everybody knows the amount of compute and network throughput that the model requires.

I don't think that's right. The, the, all the big stories about AI right now is how many data centers that we're gonna have to build. And I think EVPF can play a huge part in that.

Um, on the, the training and the buildout side, in terms of optimization, I mean, this is actually one of the reasons, uh, meta one of the original adopters of EVPF turn towards the technology. One of their first ca ca use cases was, while they're getting a lot of internet traffic from cap videos, from memes, from the messages that we're sending through WhatsApp, we need to be able to process that faster. And they turned the EVPF to do the load balancing into and out of all their data centers.

Um, and I don't have the numbers for meta, but, uh, one of the users of C'S layer four load balancer, uh, was able to reduce CPU consumption by 72 x. Right? So if you're thinking about at meta scale, if you can reduce how many load balancers you need, how many CPUs you need by 72 XI mean, you're literally saving data centers right now.

So, right. The, the performance, the efficiency improvements is igon, I think gonna have a massive impact, um, on the use, uh, on the, on the training of AI models. In fact, uh, another example, Brendan, Greg, one of the early adopters, um, of EBPF doing a lot of performance stuff at Netflix, he actually just moved to open ai, you know, to do compute optimization at the largest scale.

So I think EBPF is gonna have a huge impact on both the performance side, but then also actually, like on the inference on the downstream side too, I think everybody's concerned about security, these models, what are they actually doing? And EVPF, because it's running in the kernel, can give you granular insight into everything the application at your model is doing. So I think it's both on like the efficiency side and also the observability security side too.

Eeb PF is a huge role to play in the whole AI ecosystem. Do you think that as we implement EBPF more broadly, that the manage, it might fundamentally change because, well, historically we had all these different apps, whether they were networking, security observability, or to your point, AI running in user space and required different people and different tools. But if more things are running at the kernel level, can we converge the management of these things a little bit more?

Yeah, definitely. I think that's one of the exciting things about EBPF. Um, when I was kind of talking, uh, at the beginning when we were speaking, right?

EBPF was me 11 years ago, and at the time you only had 4K instructions. Well, now you can do, uh, a million instruction programs and you can do tail calls. EBPF is now turning complete.

And kind of like what we're seeing is the progression of things from user space into the kernel. I think the parallel that I would give is like people used to wrote, right, their own user space, uh, TCP ic, uh, TCP IP implementations, right? And then eventually we kind of realized, you know, every everybody's gonna do this.

It makes sense to kind of move it into the kernel. And I think that's the exciting thing about eeb PF actually is because the whole Linux kernel needs to stay stable for a long time. If it's there, it's kind of deployed on billions of devices worldwide, from servers to cell phones to satellites.

Uh, so we, they take very seriously what they're actually putting into the Linux kernel. Now with EBPF, what we're able to do is to experiment. What are things that, um, we're, we're not sure if they'll work out, if they should be in the mainline kernel.

What if I need to add this new functionality? You can start move some of those user space things into the kernel and experiment again. Um, and then if those things make sense, you can actually, you know, merge those into the kernel.

And so I think we're gonna see definitely progression of things that used to be running user space, uh, moving to kernel space, but it really depends on the application where that makes sense. So for us in cilium, uh, one thing that we're seeing right now is like, originally it was just kinda like layer three, layer four networking, but we're actually seeing some of the layer seven networking, including things like HTTP parsers moving into the kernel. So I think that's kind of cool.

Or another really exciting area right now is scheduling. Uh, the kernel layer has, has had the same scheduler, uh, for a really long time now, and it, it works really well for a wide variety of workloads. But if you have something like, uh, very IO intensive thing, or you have gaming where you really can compare it about latency, how do you write a new scheduler?

And now you can do that with EBPF, and people are seeing huge performance improvements for specific types of workloads because they can write new schedulers tailored to those workloads with EBPF. Mm-hmm. Um, so where do we go from here?

What's next for A BPF? What should people be looking forward to? Yeah, so, uh, the whole ecosystem is progressing all the time.

I think the exciting thing for me right now is if you look at this report, um, a lot of these things, you know, the, the companies that I listed off are these, you know, internet scale companies that were like the original early adopters, VVPF and seeing the, the benefits of that. I think what's we're gonna see next is that trickling down to a lot of other projects and products a across the whole ecosystem in terms and affecting a lot more companies across a wide variety of industries. I mean, the example that I always give is like, if you have an Android phone right now you're using eeb PF, right?

So that's already billions of devices. Um, but I think it's tripling down to more of the technologies, right? Anybody using cilium is benefiting from Eeb PF, so that's everybody running, uh, cluster with dataplane V two on GKE, every single new cluster on Azure, um, you know, lots of other managed cloud providers, if you're using, uh, Datadog and their security product, uh, you're using EBPF.

And so I think what's gonna be interesting kind of coming up is a lot of people are gonna be, you know, maybe looking under the hood of their technology and suddenly realizing, oh yeah, I'm using eeb PF two and I'm getting these benefits of, in terms of increased performance, better profiling, uh, more granular observability, more hardened security systems. And they're not gonna realize like the magic behind that is eeb pf you know, it's kind like this foundational technology. It's kind of coming this strategic platform, uh, that a lot of companies are building on.

Um, and I think these benefits that were highlighted in their report, um, are gonna be coming to a lot more companies whether or not they realize it's because of EVPF or not, um, through the projects and the products and the tools and the technologies that are building on top of EVPF. To your point about that, should I as an IT organization be conscious in driving the fact upgrade to the latest versions of Linux that support EBPF and taking it a step further, should I be asking all my providers of tools about what their roadmap is for this? Because, you know, it has some material benefits to me.

Yeah, I think that's definitely a key question to ask. It's, you know, like the cost benefit analysis. Um, you're always asking in terms of writing your own EVPF programs, I think it's like, the question is, do we already have our own kernel team at where we need to fine tune things?

And you can answer yes no on that question then that's essentially what you're doing with eeb PF is tinkering with the kernel. So are we as a team prepared to, to doing that? Do and support that?

If the answer is yes, then you know, EEB PF is a great tool. If it's not, then you're probably looking at your vendors, your service providers to do that for you. And then in terms of like updating to latest, um, kernel version, obviously there's lots of benefits, but it's what are you actually trying to do?

Are you most concerned about networking and wanna upgrade for something like Net kit? Are we concerned about security? So we want to have better support for something like, um, BPF tokens to manage the permissions of eeb PF, you know, do we want to upgrade to, uh, SX for changing our scheduling and looking at like what your specific use cases are?

Or we'll inform essentially what kernel version you need to move up to to have those, um, things from EPPF. And then on the kinda like tooling and product side, I think like what companies are looking for is like, they're not looking for, I wouldn't look, I wouldn't think of eeb PF as a checklist technology, like check, like yes has EBPF in the same way you'd be like, yes, check has like WAF wa uh, like WAF functionality, you know, um, EEB PF is a technology, it's not a solution, it's not a business outcome. Companies are looking for solutions and big business outcomes.

So I would evaluate solutions, uh, based on whether you're achieving the goals that you're actually trying to do now. And my belief, and if you look at this report too, like EBPF is gonna be underneath those tools, those products, those services that are gonna help you achieve the business outcomes that you want to because of the performance and the flexibility benefits that it has. But, uh, whether that's actually checklist criteria, I would actually say no.

But I think those types of tools and services and products are gonna rise to the top because of the benefits that Eeb PF gives 'em under underneath. It's really becoming kind of a strategic platform. So I think you will see it come out in a lot of projects.

Um, but whether it's a checklist item, I would actually say no, You're a maintainer. So what is the, what do you need as a maintainer? Are you getting enough con contributions from folks?

Do you need, I don't know, maybe just people to work on documentation, but, um, I'm asking this question because in the age of ai, it seems like we're writing more code than ever. So are people contributing more code or what's going on from your perspective? Yeah, um, I think it really depends on what type of project you're looking at.

Um, how, how that's coming to the surface. I think the links kernel, uh, as a mailing driven, a deep deeply technical project, um, has their kind of own thing. I, I would say one actually super interesting use case from the DPF subsystem is actually the first one to implement AI reviewers.

Um, so if you submit a patch to the BP BPF subsystem, it's actually reviewed by ai, um, before it goes to external maintainers. And I think this is actually a really big benefit for open source maintainers, right? It's getting the code that's being contributed, making sure it's in the right format, it's the free of bugs or as many bugs as you can think.

It's actually following the guidelines. And then when you actually have the human in the loop, you know, that's the really expensive time, the maintainers time, uh, you know, it's already kind of free of a lot of these bugs and formatting and linting things. So I think that's like a huge boost for like a, how AI can help the open source ecosystem.

And it's cool to see that the EBPF subsystem is like at the forefront of that. 'cause I think it'll help move the technology even faster, um, forward together. But then like what eeb PF is technology actually needs.

Um, I would say the most important thing for us is like hearing all the different use cases. You know, people contribute patches upstream, but like what, like EBPF is such a powerful and flexible technology that we can't think of all the different ways that people are using it. You know, I, I listed, you know, networking, profiling, observability security, now it's going into scheduling human, uh, it's going into devices, into hardware.

There's so many different use cases. Um, we want, we need to hear about them so we can make sure that those use case cases are supported by BPF. Alright, cool.

Well, folks, you heard it here. EBPF, maybe it's time has finally come because it's starting to show up in a lot more places and I think the average IT organization's gonna gain a lot more experience with it in the months ahead than they have in the last decade. For sure.

Bill, thanks for being on the show. Yeah, thanks for having me. All right.

And back to you guys in the street.