Liz Rice – Container Images: Small is Beautiful

Hi, my name is Liz Rice. I look after open source engineering at Aqua Security. And Aqua Security helps enterprises — we have products to help enterprises with securing their cloud native deployment's.

I'm also involved with the CNCF. I chair the Technical Oversight Committee of the CNCF. And so I'm pretty embedded in the world of cloud native and containers.

And today, I wanted to talk about container images and, in particular, why small container images are a really good thing. If we're going to talk about why small container images are good. It's a good idea if we dive into what we find inside a container image in the first place, then we can talk about why small images are better and how you can go about making your images smaller than perhaps they are today.

So, what's inside a container image? I'm going to guess that most of you have some familiarity with using tools like Docker to run containers. So, you're used to the idea of Docker poll to get a hold of an image that you can then run.

Inside that container image, there's basically two sets of information. There's a root filesystem and some configuration information. When you run a container based on that container image, the root filesystem populates the filesystem that you can see from inside the container.

So if you're running a container, and you look in root using something like ls, you'd see the set of directories and files that came from that container image root filesystem. And then the configuration information is information about how that container should run. So things like whether there were any resource limits set up using Cgroups.

Maybe there are some environment variables that it'd find. And that information is all held in the configuration part of the container image. Now, you're used to the idea of doing Docker poll and maybe Docker inspect that you can use to see information about a container image, but it's pretty hard to actually get inside the contents of a container image.

So, I'm going to use some tools to take a Docker image and convert it into OCI format, OCI is the open containers initiative, and it standardizes these formats for container images. So I'm gonna use a tool called Skopeo to take a Docker image and convert it into what's called the distribution format for the OCI. That format is what component like Container D would get from an OCI and compliant registry.

That format is still compressed and relatively opaque. So I'm going to use another tool called a umoci to unpack my image into what's called a runtime bundle. And it's the runtime bundle that's a tool like runC would use to actually instantiate a container.

So, let's have a look at some OCI images. OK, so I'm gonna start by using Skopeo to copy. I'm gonna use that Ubuntu image as a pretty familiar image that we may all have used before.

And I'm copying that into OCI format. And I wanted it to live in a directory called Ubuntu, and I'm gonna give it the tag latest. So that's just gonna get hold of that image from Docker registry is copying those blobs.

Now, if I look inside that Ubuntu directory, I've got subdirectory called blobs inside that there's just one directory called SHA256. And inside there, I have some files. Let's just take a look at the file type of one of those, as an example.

I'll use that first one and we can see that it's gzip compress data. This is, these sort of layers of the file system in compressed format. So still pretty hard for us to see exactly what's inside.

But now I can use in umoci to unpack image. I'm going to take the Ubuntu image, which is typed latest. And I'm going to convert that into a runtime bundle and I'm going to call that open to bundle.

And if I now look inside the Ubuntu bundle that we've just created, we can say some config information and a root filesystem. And let's look inside that root filesystem and we can see that set of directories. Just what you would expect to see if you were running an Ubuntu VM.

It would it would look exactly like that set of directories. We can also take a quick look at the config information. And you can see things like the environment variables and the capabilities that are going to be granted to this container.

This is all that configuration information about how to run this container. So, let's use runC to actually instantiate this container. So I'm going to go into that directory and I'll use runC to run this container on like, let's say, bash.

Now, on inside a container, we can kind of see that the prompt has changed. I can confirm that it is a container. By looking at the running processes, I'm not seeing anything across the whole virtual machine.

I'm just seeing these limited set of processes from my container. And if I list what's in the redirectory, it's exactly what we saw earlier in that root filesystem. That's how we get the root filesystem that's visible from inside the container.

OK, so switch back to my slides. Say, the size of the image that we retrieve from Docker or in a OCI format is largely determined by the contents of that root filesystem. The the more files there are inside that root filesystem.

The bigger the image is inevitably going to be. Taking a few examples and that Ubuntu image that we just used is around 74 megabytes. That's quite a substantial image.

I'm sure many of you will have come across the Alpine Linux distribution, which is designed to be smaller. It's actually less than 10% the size of Ubuntu. And if we use those as base images for applications, it makes a real difference to the size of those applications container images.

And NGINX, based on Debian, is like 127 megabytes. The version based on Alpine is around a fifth of the size. So that can make a real difference to what the size of the image that we need to move around.

So, why do I care? What's so good about having small images? One really important aspect of this is performance.

We need to pull that image from a registry to the machine where it's going to run. That involves network transfer. The laws of physics determine that it will take a finite amount of time to move data from the registry to where it's going to run.

And it will take longer if the images large. Bear in mind that you might have to do that over and over again. If you've got a cluster of machines that you're running containers on, that image could be required on any one of those nights.

Perhaps all of those nights. And that's data transfer, every time you need to pull that image. " And that ensures that you always have the latest version, whatever note you're running on.

So, the size of an image really makes a huge difference to the performance. The speed with which we can get that image onto the machine it's needed on. But there's also a security implication for — the size of an image has implications on security.

Basically, two possibilities for insecure container images. One possibility is that there's something insecure about the configuration. Now, I'm actually going to not really talk about that today because all that configuration information can be overridden at runtime.

So anything that you find insecure about the configuration of an image you can deal with by overriding it at runtime. The other possible security flaw in a container image is that perhaps it contains some insecurities within the files in that root filesystem. In particular, are there any known exploitable vulnerabilities?

Typically we find these vulnerabilities using a vulnerability scanner for container images. So one scanner is Trivy that my team actually build and maintain. There are other scanners available.

And what they're really all doing is looking at the packages installed into the filesystem of a container image and comparing it against a database of known vulnerabilities. Some of these vulnerabilities are really famous. Sometimes they get logos.

They get names like ShellShock and Heartbleed. And if you're running with a critical vulnerability like that, it is kind of an open door for an attacker. So you really want to make sure you're not running with vulnerabilities.

Now, it's kind of a statistical truism that the less code you have, the less complex it is and, therefore, the less likely it is to have vulnerabilities, whether they're known or not known yet. It also sort of stands to reason that the more packages you have installed in an image, the more likely it is that one of those packages has a known vulnerability. Say.

Also, bear in mind that vulnerabilities get found in existing code. They can be found in code that was written a decade ago. So, you should be scanning your images on a regular basis to find these vulnerabilities.

And the smaller the image is the fewer packages it contains, the less code it contains, the less the likelihood is that it includes some of these vulnerabilities. So statistically speaking, the smaller your image is, the less likely it is to have known vulnerabilities or even vulnerabilities that hadn't been uncovered yet. So, smaller images are better from a security point of view, and they're better from a performance point of view because of the speed of transferring those images.

What kind of things do you perhaps have in your container images today that maybe you don't really need to be there? Say, one thing that we maybe don't see so often but could be in container images today is an SSH Server. No reason to have an SSH Server on a container or in a container image.

You can always get into a container using tools like docker exec or kubectl exec. So there's no there's no particular reason to have SSH running inside a container. I would actually advise not to be exacting into your containers on anything other than a very unusual basis, and really you should be treating your containers as immutable.

If you need to change something about a container, you should rebuild the image and redeploy new versions of those containers. So treat your containers as immutable. Don't be exacting into them and changing the contents of those running containers.

Another thing that you probably don't need inside your containers is logging specific code. Containers aren't designed to take an output that's written to stand it out or send it and write back to system wide logging. So you don't need logging demons running inside containers.

Cron is another example of something you probably don't need. You may very well have jobs that you want to run on a scheduled basis that all containerized, but the cron aspect of that doesn't need to be inside the container. You can use your orchestrator to schedule the jobs and have them just run as a short running job at the appropriate schedule.

And another thing that you maybe don't need inside a container is a package manager. I've just said that we should be tracing our containers as a meatball. So, why would you want to be adding and removing packages inside those containers?

Well, it can be very useful to be able to add packages at bill time. I hope you're not doing at runtime. So maybe we need a packager inside the container during that build step.

Some other things to consider that maybe we don't need inside containers. How about shells? A shell can be a hugely useful tool for an attacker that gets into your container.

So it might be nice to remove shells from container images so the attackers can take advantage of them. But, depending on your application, you might need to shell. It's totally legitimate to have shell scripts running inside containers.

So removing the shell might be a step too far depending on your application. If you do remove a shell from a container image. Bear in mind that you need to define the entry point in the command in the non-shell format.

Otherwise you would have a dependency on that shell. And then how about the packages that are installed into your container images? You really want to make sure that you only have the packages that your application really needs.

Some examples of packages that you maybe don't need and that maybe should ring an alarm bell. I would question why you need curl in a container image if you're going to treat that container as immutable, why would you be downloading code using curl? SSH, again, no real reason to be SSH in either to or from a container, in most circumstances.

And then finally, how about executable utilities? Do you really need your container to be able to list its own contents or change the permissions inside a running container? Maybe you don't really need that code at all for your application.

That said, if you're using a container image based on Alpine, Alpine is itself based on a busybox and busybox has essentially just one executable symbolically linked for all of these, the binaries like Atlas that exist in /bin. OK, so we've seen a whole set of things that probably don't need to exist in your container images or might not need to exist in your images depending on your application. Seems like it might be pretty painful to actually go through all your container images and remove these different items individually.

So, what's a more practical way of making images smaller? One option is a tool called dockerslim. And this is — it inspects your image, removes the unused files that figures out aren't really needed in that image and creates a smaller, slim version of that image.

However, dockerslim removes package information and the package information is really what the vulnerability scanner needs to see what packages are installed and therefore whether or not those packages have vulnerabilities. So, let's take a look at dockerslim in action. So I need to exit out of that container I was running before and get back to the previous directory.

I can run dockerslim in interactive mode and I'm gonna build based on targets. Let's take NGINEX, we'll take the latest version of NGINEX and we will creates a slimmer version of NGINEX, and this will just take a few seconds to investigate what's inside NGINEX and figure out what files and packages can be removed. OK.

And once that's finished, I can exit. And now if I do a — list the NGINEX images, we can say that that slim version that I've just created is dramatically smaller even than the Alpine-based version of NGINEX. So, it's kind of half the size.

So, that seems pretty, pretty good. Pretty encouraging. Now.

Let me show you what I meant about removing the package information. So if I run the the original version of NGINEX. I'll just run NGINEX and I'm gonna list the packages that are installed.

So it has the APT package manager inside that image and we can see this, dozens of packages that APT finds inside that original pre-slimmed version of NGINEX. If I tried to do the same thing on the slim version, it's not going to work because that APT package manager is one of the things that dockerslim recognized wasn't needed to run NGINEX, so it's been removed. That doesn't necessarily mean that all the packages have been removed, just that the package manager has been removed.

Now, the package information exists inside a var/lib/d package. So if I go to the original one and I list what's inside th ere. We can say and some directories as there's information there, like status.

This is the information that vulnerability scanners will use to determine which packages are contained inside image and therefore whether or not they're vulnerable or not. I can't just list what's inside the slim version. Let's just try it.

I can't do it because that executable, ls is one of the things that dockerslim has removed. But what I could do is do what we did earlier. Use Skopeo and umoci to convert NGINEX slim image into OCI format and then into the runtime bundle.

And then we can look at the filesystem. And I actually already did this. So I have an NGINEX slim bundle and inside that would find a root filesystem.

There is a var directory. But there's no var/lib. There's only var cache, var login, var run.

So this package information has been removed from the image and because of that a vulnerability scanner wouldn't be able to determine whether or not this slimmed down image has any known vulnerabilities or not. Alright. Another option for small images is distroless.

This is a project by Google to create a set of minimal images for different languages. They take Debian images. They remove unnecessary binaries.

They remove the package manager. They don't remove packages. And also, they don't remove all of those var/lib/d package information, say.

Because it leaves that information in place destroyed as images can be scanned by at least a certain vulnerability scanners. And what the distroless project recommends is using these distroless versions of the images as the final stage in a multistage build. So I've taken this example, ducker file straight from the distroless project.

It's showing an example of building Java application. So the first five lines of this Docker file use the Java toolchain to build that application. And then the last few lines start from that distroless image, that base image for Java.

Copies the application into this distroless image and so we end up with a distroless image and just the application and what it needs in order to run. We don't need the whole build toolchain. And there's examples of distroless for a variety of different languages.

And again, we can take a look at the contents of distroless image. So, I have also used Skopeo and umoci to unpack this distroless image. So let's have a look at inside that.

This is the Java bundle. If I look inside the root filesystem here, we can actually see that is v ar/lib/d package information. It's actually all contained inside this status, the information.

This is sufficient for certainly Trivy can use this information to determine which packages are installed inside this information. So, you can build these small images based on distroless without sacrificing the ability to scan for vulnerabilities. Now, the most extreme small image that we can use is the scratch image.

Scratch is kind of a base image, is not really even a base image. It's more a reserved word in Docker, representing an empty image with nothing imagined that root filesystem with no files, no directories, nothing. And then you can use this in Dockerfiles in the FROM command so you can start with FROM scratch and add in whatever files your application contain is going to need.

My example there is imagining having a single executable called hello and just copying that single file into the container image. If you don't need any packages, you can't have any known package vulnerabilities. You know, they can't exist if there are no packages.

So, using scratch container as a basis can be a really effective way of of avoiding any of these known package vulnerabilities, particularly if you're using a language that compiles to standalone executables. If you're using languages like Go or C, scratch could be a really great option. One "gotcha" is that if you are using secure connections, TLS connections — sometimes I use SSL connections — your binary might need to be able to verify certificates with the component that is setting up connections with.

And in order to verify certificates, it will need certificate authority information. So, you would need to add the certificate authority files into your scratch-based image. That is one thing that people do sometimes get caught out by.

But scratch can be a hugely valuable way of getting tiny images. You might want to consider using it in a multistage build, where you have a first stage similar to what we saw with distroless that uses the language toolchain in that first build step, and then has a final stage that starts from scratch and copies the executable and any other dependencies, any other necessary files into the scratch or into an image based on scratch, starting from nothing. So, scratch is probably the ultimate way of making the smallest possible container image for your application, but size isn't the only thing you should consider.

There are other things you should take into account. In particular, you don't want to be going for the smallest possible container image at the cost of making it hard for developers to use those images. You want to know that if you're using a small base image that it is well maintained and if it's got any packages at all, you want to make sure that those packages don't have vulnerabilities.

So do bear in mind that that base image contains code. And that code can be vulnerable. You need to be able to scan that base image.

So, my recommendation is there isn't just one possible root for building small contained images, your best options depend quite significantly on your application language. For as binary languages, more compiled languages that compile into standalone binaries, scratch could be a good option. You may want to look at other smaller base images like distroless, possibly like Alpine, as the starting point for other interpreted languages.

It is probably a good idea to try to find images that aren't based on the entire full distribution. For example, if you take a desktop distribution, there's going to be a whole lot of things that your container doesn't need, and that's wasteful. So start from small and appropriate base images for your application and for your language.

Consider using multistage builds to avoid having those sort of toolchain requirements in the final container. And do you make sure that whatever base image you're using, make sure that your vulnerability scanner supports it, knows how to interpret the package information inside it and can let you know about any significant vulnerabilities in any packages. So, I hope that's giving you some insight into container images and their contents and how to make small but relatively secure — I'm never going to say they are completely secure because new vulnerabilities get discovered all the time.

But hopefully those aren't helpful tips for making smaller, usable, pragmatic images. If you want to get more information about best practices for building secure container images, there is additional information in my book, which recently came out, published by O'Reilly, about container security. tech, from there you will find a link to the Aqua Security site, where Aqua has sponsored the book and you can get a free electronic copy for the price of your contact details.

So, do download the book if you want to dive into more information about building secure container images and other aspects of container security. Thank you very much.