Enhancing Cybersecurity in Linux with Chainguard’s Jason Hall

This is Textron tv. Hey guys, thanks for the throw. We're here with Jason Hall as a principal engineer for Chain Guard, and we're talking about the anniversary of the inaugural launch of Wolfie, which was a new distribution of Linux that these guys created that's a little more secure and maybe easier to deal with, but Jason will get into all that stuff in a minute.

Jason, welcome to the show. Hi. Thanks for having me.

For those that don't know what wolfie is exactly, explain what makes it different than other distributions. I know you guys have called it the UN distribution, but, um, put some perspective around this. Yeah, so the, the distribution name, uh, is mainly, um, uh, a thing we say because it doesn't include a kernel.

Uh, we focus in wolfie on container-based, uh, the container-based ecosystem and use case, um, where if you're putting things in a container, you're actually using the host machines kernel to schedule stuff. So, um, wolfie does not currently include a kernel. That's why we sort of call it the, the tro.

The main focus of wolfie is, um, lowering the number of vulnerabilities that you experience as an end user of the, of the distro. Um, there's a few ways that we do it. One is to make packages as minimal as possible.

That's by far the most, uh, the most easiest way to have as, as few vulnerabilities as possible is to have as little code as possible running at any given time. Uh, the base wolfy base image that we provide contains almost nothing. It's very, very small, sort of like alpine in that way.

Um, and every package is designed to be as minimal as possible so that you have as, as granular as possible, uh, uh, packages when you install 'em. Um, the other way that we reduce vulnerabilities in images or in, uh, in, in the environment is to update packages as quickly as possible. So first, we give you as little as possible to get as, uh, as little as possible to get the job done, and then we also keep those packages up to date as quickly as we can.

Um, I can go into a lot more about how we do that, but the short answer is automation. We automate the crap out of, uh, updating packages. As soon as there is an upstream, um, release, we will detect that and package it and release it.

Uh, usually within, uh, an hour or two, we have some, some data about how quickly we can do that, and it's, you know, normally within an hour or two of the upstream release, maybe a day or two, no, it's days. Sorry, it's a day. So at, at its core, however, it's basically smaller and more frequently updated to, uh, prevent, uh, exploits that the bad guys are gonna find ways to kind of worm their way into and do something nefarious.

Um, it kind of sounds, shall we say, um, intuitively obvious thing to do. So what was the hard part about this and why didn't we do it before? Um, yeah, I think the, the hard part about it is balancing stability with these update speeds.

Um, a lot of distros, I mean, every, you know, every distro is doing a great thing, is doing a great service to, to the world by, by producing and packaging these things for people to use. Um, uh, but, and a lot of them, uh, I don't wanna say overfocus on stability, but stability is more important than updates. Stability, uh, is, uh, more important than being able to get updates out as fast as possible.

This is also sort of an artifact of having a very large course packages, um, when you install, uh, a package in other distros, it contains a lot and therefore there's a lot that can break on an upgrade. Uh, in wolfie, we try to have as small as possible packages and as granular as possible so that each of those can be, uh, relatively focused and, and, and relatively, uh, you have a relatively high amount of confidence that those are working like you'd expect. The other thing that, that, uh, wolfie, um, uh, I think really focuses on is the container ecosystem where we don't have to, we don't have as much of a focus on desktop software or, um, even, like I said, the kernel, the kernel is a huge, a huge piece of code that's a, a gigantic massive, um, thing that we don't have, uh, we don't have to worry about it all.

So, um, by focusing on container workloads, um, uh, we can, we can provide very small focused packages and keep them up to date as fast as possible. Um, another thing is, uh, because of the container ecosystem that we're living in, upgrades in place are very rare. Normally in container workloads, you're not sort of doing an upgrade of the curl package or of the bash package.

You're just destroying that container and creating a new one with the, with the, uh, updated package. So we have a lot of simplicity in terms of, we don't have to worry so much about the upgrade path from an old version to a new version. You're just, we assume you're going to destroy that container completely and recreate it anew with, uh, with updated packages.

So those are a few ways that we, um, are able to provide this, uh, differentiation by focusing on a particular, uh, space in the, in the computing sort of landscape. You mentioned not focusing on the desktop. Are there any other important functions that maybe we're not gonna see because it is smaller?

Or have you just kind of narrowed it down to the bits that we really need to run? Yeah, I'd say we, uh, I should, I should clarify. It's not that we don't think we'll ever get there, but that's just not where we're focused right now.

So we're, we're focused on the container ecosystem now. I have no, uh, I have no idea. We've been around for only one year.

Right. And we're already where we are. I have no idea where we'll be in another year or two, or three or five.

We may get into desktop software, we may get into iot, we may get into install. We may have a kernel. I have no, you know, I, i, I don't pretend to understand where will this will go in the future, um, given the, uh, the sort of meteoric rise we've seen even over the last year.

But definitely I'm not ruling it out either. So define meteoric a little bit. 'cause you know, we don't know how many folks are using what open source software these days, but we have certain metrics we track.

So what are you seeing? Yeah, um, we've seen a, a lot of, uh, uptake from users. Um, we provide, uh, an image that anybody can use for free.

Um, we've seen that cropping up in a lot of, uh, in a lot of places, which has been really exciting. Um, we've released, I think, oh, I have old data, which has 1300 packages. I'm sure it's more like 1500, 1600 packages today.

Um, and we're, the repo is checking this morning, we're approaching 10,000 prs, uh, which is really gonna be, uh, incredible. A lot of this comes from both like human contributors, human, uh, uh, you know, contributors adding new packages and updating packages. But even more so lately, the automation that I was talking about to keep these things up to date, that's really sort of, uh, kind of juicing the numbers on the, uh, the number of prs that we're creating.

'cause the robots are out creating them for us. Um, but we've seen a lot of, a lot of, uh, external contributions from, from non-chain guard folks, uh, adding new packages, updating new packages, and, uh, yeah, it's been really, it's been really great to see the sort of, uh, uh, uptake in the community. Who's driving adoption of this in the average enterprise or wherever you're seeing it rolled out, who's waking up in the morning and going, I know today's the day I gotta go get me some wolfing.

Yeah. Um, the, the main sort of driver for it so far has been chain guard's, own chain guard images. Um, all of all of our images are built on top of wolfie packages.

Um, and we produce those both, uh, for anybody to use the latest version of those images for free. And we, uh, have customers that buy, uh, older versions and support for those, for those images. That's been the main driver of adoption of wolfie so far.

But we've seen, um, all kinds of uptake in interesting ways. Uh, the community meeting that we have, uh, for wolfie, uh, most recently there was a, a, a team from, uh, dagger, um, talking about how you can use Dagger to create images from Wolfie packages. Um, we've had folks from, uh, suse, we've had folks from, uh, I'm forgetting off the top of my head, but we've had a ton of, uh, external contributors come and show us what kind of cool stuff they're building, uh, with Wolfie.

It's been really awesome. Is there a security aspect to this as well, because it's a smaller attack surface, so are those folks involved in this conversation, or where do they fit? Yeah, absolutely.

That's the, you know, like I said, that's the, the main focus of the distro is to minimize the attack surface as much as possible. Um, a lot of the folks who are interested in chamber guard images have been interested from that angle, both in terms of, you know, reducing the literal attack surface and in terms of just quieting down the scanners and, and, you know, not showing 10,000 vulnerabilities in all of their images, you can switch to a chain guard image and see like zero or one or, you know, usually zero. Um, so, so that's definitely been a part of it that the, uh, and again, the way we do that is to just update these things as quickly as possible to, um, uh, even if there is a vulnerability detected in your current image or in your current package within a a day, maybe you should be able to update and get rid of that.

So, um, that's definitely been a, a big driver of usage for, for wolfie and, uh, and the ecosystem so far. Of course, there's a lot of talk these days about what do we do to better secure our software supply chains. Um, is this part of that conversation and fundamentally, are our software supply chain's kind of broken at the moment?

Yeah, there's, there's a lot to fix there. Turns out there's a lot of, a lot of angles, uh, that, uh, we have been ignoring for I think too long. Um, definitely this is, this is our sort of solution or one of the solutions to one of the many problems in the supply chain space.

Um, getting people updates very quickly is, uh, important. I think a lot of people don't realize how much time there is between, uh, an upstream fix. Like you're depending on, you know, something foundational like Curl or bash or even just, you know, get, or any tool you use day to day, there's an upstream fix for that after a, after a vulnerability's found in it.

And then it can be days, weeks, sometimes months before that's available to you as a consumer of it. And even then, once it's available to you, now it's up to you to go and update that image and, you know, roll it out to your actual, uh, uh, fleet of, uh, production. So I think Wolfe's, uh, Wolfe's goal is, is in taking that amount of time, understanding that amount of time, and then minimizing it as much as possible so that, um, we can't really do anything about how quickly you update your software, but we can at least give you updates as quickly as possible.

Um, another thing that Wolfie does really well, I think at this in the, uh, sort of area of supply chain security innovation is the amount of visibility into the package builds and the, the provenance of those and the image builds is I think really sort of state of the art. Um, you should be able to reproduce the package that we build from source yourself. If you, if you don't trust us, you should be able to build it yourself and get the same result.

Um, which I think is, is a sort of a, a, a constant struggle in the, in the world of building packages, is like, how do I trust this? How do I know this came from this source? How do I, how would I prove it?

Um, and that's something we're also sort of really focused on. 'cause even better than trusting us is not having to trust us, right? Uh, you should be able to get this stuff yourself.

Is there any thought to contributing this to any of the larger Linux consortiums that are out there? There seems to be a lot of bodies that are working on various open source projects. Where do you fit in the larger ecosystem?

Yeah, I mean there's, there's a ton of other projects, right? There's, we are not the, the first distro and we won't be the last. Um, I think that, um, there, you know, there are plenty of Distros doing their own thing and, and they all seem to be happy doing it wherever they're doing it.

What's your best advice to folks about how to get started with all this then? Is it seems like a lot of places people are going, I understand that I gotta do something about my software supply chain, but I don't know where to get started. Do I start with, uh, Lenox distro somewhere else?

Or what's the path to success? Yeah, that's, that's actually one of the, one of my favorite things about this as a, as a project is it's really easy to get started. Um, if you are building an image with a Docker file today, as a lot of people are, um, that first line of the Docker file is usually from something right?

From Debbie and from from go from Alpine. dev/chain guard slash wolfie base, and now you will be opted into wolfie like you, that's that one line changes all it takes to, um, to get wolfie packages. And, uh, the benefit is immediate.

As soon as you start building that image, you get up-to-date packages as quickly as we are able to produce them. Um, so that's, that's a real, um, benefit and sort of, uh, it makes a really good booth demo, right? Like, it makes it really easy to say like, well, what are you using today?

Let's change this, change this one line Docker build, and now you are, you have done it, you are updated, you are in Wolfie, and as soon as there is a fix for this vulnerability, you will get it also. Well, folks, you heard it here, it's a simple concept. The more old code there is lying around, the harder it is to protect it.

And so shrink the base and go from there. Hey Jason, thanks for being on the show. Absolutely.

Thanks for having me. All right. And back to you guys in the studio.