Cora Iberkleid & Andreas Evers – Creating Production Ready Containers with Cloud Native Buildpacks

I'm very happy to welcome a Cora Iberkleid, developer advocate from VMware, one of our sponsors, and thank you, VMware, for sponsoring Commit and Andreas Evers, senior solutions architect, also from VMware. Their talk is titled "Creating Production Ready Containers with Cloud Native Build Packs" Cloud native apps tend to be based on container images, which have to be frequently rebuilt. Cora and Andreas take us through the evolution of building container images as they zero in on the current state of the art, which will demonstrate the use of.

This is great stuff. Let's dive in. And don't forget that if you have any questions or comments, you can chat with the speakers and other attendees during and after the talk using the chat function.

All right, so thanks for joining our talk. Andreas I will be talking about creating production ready containers using a project called Cloud native built products that you can also use within GitLab, and we'll show you how. My name is Cora Iberkleid.

I'm a developer advocate at VMware. And I'm Andreas Evers. I'm a senior solutions architect also for VMware.

Great. So today we're going to be giving a basic intro to different ways that are common for building images. And then we'll walk you through the cloud native buildpacks project in particular.

We'll do a few demos both outside of GitLab so you can see how you can use this locally at your command line. And then we'll show you how it works within GitLab. And then we'll cover some additional features of the Build packs project and we'll wrap up with a summary.

So with that, go ahead Andreas with the intro. All right, thanks a lot for that, Cora. Let's jump right in.

So this should look pretty familiar to people. And a lot of organizations who started with code and a git repository in your CI pipeline built an application artifact after which you deploy that artifact into an environment which is managed by an operator agency. However, Kubernetes is quickly becoming the platform of choice and shifting responsibilities to the left into the CI CD pipeline.

So responsibilities such as building an image, including any middleware and operating system, matching the entire stack as time goes by, making sure that everything is secure and everything is done according to compliance and accountability standards. So this really begs the question better the sweat and tears that we were experiencing farther on the right of the flow are now going to shift left, or maybe we can prevent that increased complexity and make it an easier process for ourselves. So that's really the question at hand.

How are we going to go about building that image? Let's take a look at some common approaches that people are using for building images. And let's start with the most common way of building an image through a Docker file.

But first of all, what is a Docker file? A Docker file is basically a script in which you describe exactly how your app needs to be packaged into an image. In fact, if you use the templates from GitLab, you might already get Docker files out of the box.

Or, of course, there's a good chance you might have already written your own Docker files. And as you can see, these Docker files can vary a lot. And you can basically write whatever you want and it can be very, very powerful.

But with great power, as Spiderman's uncle once said, comes great responsibility. And all of that responsibility is on you. Responsibility of figuring out what the best practices are and how to make it efficient, how to make it safe, how to support the different components that you're building in.

And because it's very difficult to share this across teams or across an enterprise, it really creates this hotbed for snowflakes and or black boxes, which is difficult for the organization to manage in the long run. So we want to look for a higher level abstraction. There are a few solutions out there with each their respective pros and cons.

And for Java, one choice that's been very popular is Jib. It builds quickly. There's no local Docker dependancy and it integrates seamlessly with your Maven or Gradle workflows; where it falls a little short, though, is that it doesn't have a centralized governance mechanism.

Think about how teams would update these images, how organizations would like to have visibility on the different versions of base images running a production. Over time and scale, you're still going to find a lot of variance across your images and that's going to increase the risk of mistake. Jib and Docker files don't really help manage that risk in any way.

And that's where Buildpacks might offer a solution. Build packs have been around for a long time. They actually predate Docker and Kubernetes and are an instrumental feature of both Heroku and Cloud Foundry, two prominent platform-as-a-service solutions.

They allow you actually to run to take source repositories and turn that into a runtime environment and a consistent, repeatable, secure and audit-able way. Without the need for low level abstractions such as Docker files. Now, Herokuish aims to bring the idea of the Heroku build packs to GitLab without actually having to run the Heroku platform.

And since 2017, GitLab has a new feature called Auto Devops and it uses Herokuish and Buildpacks. And in case you're not familiar with Auto Devops, the idea is to provide a pipeline without the need for any Yaml configuration or Docker files. For the build and test sections of the Auto Devops pipeline, GitLab uses Herokuish by default.

But those original Heroku built backs from 2011 were also difficult to modernize, and they didn't have a single API standard. And that brings us to cloud native buildpacks, the natural evolution of buildpacks, the original buildpacks. This is a project that was initiated by Pivotal, which is now VMware and Heroku about two years ago to improve upon the older model of buildpacks while also leveraging the modern technology, the modern container technology actually that has evolved more recently.

Pivotal and Heroku have a nearly 10 year history of building and supporting enterprise with build backs. So all of that knowledge, all of that code can be taken and tailored to work with Kubernetes. So cloud native buildpacks is a unified standard that enables an ecosystem of build platforms and modular build packs with a goal of taking source code and producing Docker images that can then be stored in any registry of your choice and deployed on any Docker runtime.

And also, GitLab has adopted cloud native build parks in the Auto Devops pipelines and Herokuish has been deprecated in favor of cloud native buildpacks. So now, Cora will take a closer look at cloud native buildpacks and how it actually works. All right, thanks, Andreas.

OK, so I'm going to give you a bit of an overview of the Cloud Native Build Packs project first and then I'll show you a quick demo. So as an end user, the two sort of elements that you'll be interacting with most often when you're working with cloud native buildpacks is a platform and the actual build packs . And the platform is the tool that you use as an end user.

It could be a CLI, it could be a service, maybe hosted on Kubernetes. It could be a Maven or Gradle plugin. There are different platforms that will give you the ability to use build packs.

Now, the build packs are the components that actually contain the intelligence for taking source code and turning it into the layers that finally comprise your runtime image. And the idea behind these bill packs is that they're modularized and so that you can easily share them so that they're manageable, that they're composable. And the idea is to make it very, very mature from an operational point of view so that you're building images in a way that's manageable at scale, auditable, repeatable and reliable.

Now, the build packs in order to make the project easy to distribute and utilize the build packs are distributed on an image called a builder image, which acts as your build time environment and that build time environment. That build image also includes a piece of software called the Lifecycle, which contains the generic orchestration logic that any platform would have to execute in order to take advantage of buildpacks. So rather than have every platform or buildpack implementation write this generic code the cloud native build pack project provides this reference implementation and a build pack builder image can simply incorporate that and make it available to the platform.

And then there's a reference to a slimmer run image because you don't want your build tooling like you wouldn't want your JDK to be deployed to your production environment, Right. You only need a JRE. So the build time environment is one image and the runtime environment is a slimmer second image and the components of your compiled class files, for example, would be transferred from the build image to the run image to produce the image that you're going to deploy into your production environment.

So. As I mentioned, there's an ecosystem growing around this because what the project offers is the platform API and the buildpacks API, so they're really just establishing this unified standard, as Andreas mentioned, in order to support growing ecosystem around this. And so from a platform perspective, there's the ecosystem currently includes the Pack CLI.

So Pack is a CLI that gives you an imperative tool with which you can build images. And this is actually the reference platform in implementation that's provided by the cloud native buildpacks project itself. Recently, for example, Spring Boot and Spring Boot 230.

introduces support for cloud native buildpacks in its Maven and Gradle plugins. So if you're familiar with Jib, the Springboot plug ins give you a similar experience that they use as a cloud native build packs project to produce an OCI image. So with a Maven package or Gradle build, you would have an image that has all of the goodness of cloud native built packs, KPack is a service that you can install in Kubernetes and configure it declaratively that can operate at scale to build and update images and so on.

So the ecosystem is growing, which is which is nice to see. And on the builder side, there's also a growing ecosystem. Heroku and Pivotal, of course, have taken their previous set of Cloud Foundry and Heroku build packs and updated them to comply with the cloud native build packs standard.

And so you have the Heroku build packs and the Paketo build packs are the evolution of those Cloud Foundry build packs. But here there's also a growing ecosystem. Google Cloud platform recently came out with a set of build packs.

VMware Tanzu is a commercial superset of Paketo build packs. Both ecosystems are growing. And so what I'm going to demo for you is using the pack CLI and the Paketo build packs together to create an image and the Pack CLI is actually what the cloud native integration in GitLab is using.

We'll show you that in a second. OK, so in this directory, I'm just going to build a simple Go application is just a hello world application written in Go. But as a point of comparison, since Docker file is sort of one of the most common way, the oldest way to build Docker images, I wanted to draw a few comparisons to it, to how you can think about this in relation to a Docker file.

So this is a fairly decent Docker file, but it's not perfect, but not bad. And you can see it has to code blocks, starting with two front statements. So the first one is the build environment.

And so we're using an image called Golang publicly available on Docker Hub that has the tooling necessary in order to build the source code into Runnable application. And then the second code block, the second build stage, as it's called in Docker file, starts with a the a more minimal base. So in this case, we can get away with the most minimal, which is scratch, but it really depends on what language, what framework you're using.

And it copies the binary that was created during the build stage into what's going to be your runtime image and then establishes what command should be run when this image is run as a container. To execute this Docker file, you would use the Docker CLI , your Docker file and your source code would be sent up to the Docker demon. The Docker demon would run the Docker file and produce the final image.

So when we're talking about the cloud native buildpacks project, there's some comparisons we can draw. So the base image, so it would be the case, the Golang, the build image and the runtime image in this case, the scratch in this example would be coming from the build packs implementation project. And so we're using Paketo build packs for this demo.

So that already provides the builder provides the actual build image and it has a reference to the run image to use. In that builder image are all of the build packs. So the build packs are the pieces that would basically provide all of the other commands in your doctor file.

Right. All of the other knowledge and intelligence to produce an image from your particular application and the Pack platform, as well as that piece of software I mentioned called the Lifecycle that's also in the builder image together play the role of the Docker CLI and the Docker demon in order to orchestrate the production of the final image. So I'm going to show you an inspection pack allows you to inspect both builders and images.

So first I'll show you an inspection of the builder itself so you can see what's included inside of it. And you can see, for example, that this particular builder supports several frameworks, Java, Node, Go, cetera. So it works well as a default builder.

I've already set that as my default builder, in fact. And I have a Go application, so I know that it'll work. And then I can see the full list of build backs that are included in this builder.

So you can see that for every build back there's multiple, for every framework, there's multiple build parks that might participate in a build. And so there's this detection order that's established that enables the process to auto detect, which build parks to use for a particular application. There's also some more metadata here about the image that we're going to use for runtime.

And for example, I can see the stack. So this is a boon to stack for both the build time and the runtime. So with that, I'm going to.

Show you the build command. So, again, we're using Pack as a platform, which is a CLI, and all we do is call Pack a build. I don't have to specify the builder because I've said it as my default on my machine and I just provide an arbitrary name for my image.

And I've we're in the we're in the directory where the source code live. So it's going to look in the local directory and I've already pre downloaded that builder image. So I'm just saying no pull.

So I'm not pulling from the Internet to make it a little bit faster. So what you see now is the different stages. The platform is orchestrating the lifecycle stages.

And so these stages of detection, analyzing, restoring, building, exporting are the different stages of the orchestration of producing an image. Those are the generic stages. And so you can see during the detection we go through that detection order in the builder and it automatically determines which build pacts apply to my application, the analyzing and restoring stages.

I've already built this image once on my computer before, so I have the prior built image with its metadata as well as a local cache. So the process can now take advantage of both of those sources of information to make subsequent builds more efficient. So that's what's happening between analyzing and restoring.

And then the actual build is where every build pack that was detected is relevant, is executed one after the other. Right. You can see here's one build pack and then the second build pack is here.

And then the third build pack that was detected is run here. And each build back is producing layers. So the most of this logging is coming from the orchestration, the generic orchestration from the lifecycle.

But what you see inside of the build stage, that comes from the build. So this is coming from the Paketo build packs in particular. Right, these pieces of logging.

Now, once that's done, the layers that are relevant for runtime are exported to the run image and the layers that might help subsequent builds be more efficient, like the go compiler are cached locally. And finally, we have our image to use. So if I can look at my docker images, I can see go demo image was created.

Images are reproducible, meaning that all information like timestamps is all made the same so that we can get the same ID no matter when we build the image, no matter what platform we're using, as long as we're using the same builder and the same source code, we'll get the same image and. I can also build this, and rather than publish the image to my local Docker demon, I can add a dash dash published lag and it could be built directly on the Docker registry. So that's what I wanted to show you from that perspective.

So I'm going to hand it back to Andreas to show you how this works within GitLab. All right, thanks, Cora. Let me show you my screen again.

There we go. All right. So let's take a look at how this can be used inside of GitLab.

Of course, the Pack CLI command itself can be used in any pipeline to wish tow rite yourself, but Auto Devops gives us an opinionated out of the box pipeline which has provided this implementation for you. In fact, if we take a closer look at the source code behind the build stage of this Auto Devops pipeline, you'll see exactly the same Pack command that Cora used in the previous demo. By the way, it's quite useful to have access to the source code behind Auto DevOps.

It's basically just a bunch of shell scripts which aim to provide the boilerplate code at 80 percent of the pipelines out there typically duplicate. The Auto DevOps developers have written this for you, so you won't have to, and that sounds very similar to Spring Boot auto configuration if you're familiar with that framework. Manipulating that default behavior is also very straightforward.

We've got a number of variables at our disposal here. The first one enables CNB or cloud native build packs beta functionality and turns off the use of Herokuish for building images, and this is the only required flag or a variable, if you wish to use cloud native build packs in Auto Devops. The second one sets the builders you'd wish to use.

In our case, this would be the Paketo builders. With the third variable, you could pass in a specific URL where buildpacks can be found. This can be particularly useful for custom-build packs and we'll talk more about that later on.

And with the final variable, we can pass any setting to the build packs themselves. In the next demo will demonstrate Auto Devops with the use of Paketo cloud native buildpacks and the first way to use Auto Devops with Cloud native buildpacks is through use of variable configuration. And this requires having the Auto Devops setting turned down in the project's CI CD settings and making sure you don't have a GitLab CI Yaml file in your repo because you don't actually need one.

And now the second way of of using our Devops functionality within within GitLab with cloud native build packs is to still use GitLab CI Yaml file, so you are still able to use it. But in this case, it's important to explicitly inherit from the default Auto Devops configuration. Before we add in any extra configuration, we can define the same variables as the previous approach.

But now, as part of the Yaml file, as you can see here now, there's a number of things that we added here. Let's go through them. *, which is the default.

We've also added Maven settings to enable running of tests during the builds, during the Maven builds. Now this is just to demonstrate how you could run test as part of your Paketo or cloud native build packss in general using exactly the same platform as your builds. And we've also added in a custom built back here, Hello World.

And in the before script we we can add any logic on top of the Auto DevOps default behavior. And currently we clone the buildpacks sample, the samples repository and reference that in the buildpack URL, as you can see here. OK, let's take a look at the logs of this pipeline run specifically.

Now I've triggered this pipeline while Cora was talking to speed things up a bit. So if we take a look at this stage here, we can see it has succeeded, but the logs are a bit too big. So I've opened them here and try to colorize them a bit.

Sadly, there's no real plugin for this in Chrome. So you'll have to bear with me here. Let's take a look at the detection face that Cora was talking about.

As you can see, it has detected that my application in my repo is a Java application with Spring Boot, and it has selected a set of build backs that it can execute for this particular build, for this particular repo. As you can see, we're using the JVM Bellsoft Liberica buildpack, which has always has a doc link here or a link to the home page, which is very useful if you want to get more information about how the build pack works internally and you also get environment variables that you can set specifically for this build pack. And in this case, we've set it to 14 for the Java version.

And indeed, it has also propagated that variable. Then there's also the Maven build back here, which if you're if you're actually using Gradle, it will as well swap this out for the Gradle one, but all the other build backs will still remain the same. And this shows the modularity and compatibility and reuse of cloud native build packs in action.

We also have something interesting in the GitLab I the memory calculator. There we go and the memory calculator, it calculates a holistic JVM memory configuration with the goal of ensuring that the application performs well while not exceeding a container's memory limits and being recycled. We can also see that if we look at test run or tests run, that indeed, tests are being run now as part of the package.

And to finalize the Hello world custom build back, it is indeed printing successfully all of the environment variables. So basically, it's pretty straightforward to use your own custom-build packs, if you'd like. Let's go back to the slides.

Right now, Cora will take a closer look at some additional features of cloud native buildpacks. All right, thanks, Andreas. So a couple of features that I want to show you in addition to that.

So I'm going to show you inspections. We already saw an inspection on the builder. I'm going to show you how you're going to inspect an image.

And then I want to talk to you about this feature called Rebasing, which is swapping out the operating system layer. So this is an incredibly powerful security feature. And think about a situation like Heartbleed or Spectre, where there's a vulnerability in the operating system.

And as an enterprise, you might need to patch hundreds or thousands of virtual machines or container images. And so this in the case of cloud data build pack. The way that this works is so you say you have an image and a vulnerability is detected at the operating system level.

A new operating system with a patch is made available. Right. And as soon as that patch is available, you would obtain the new run image.

And depending on the build pack implementation that you're using, ideally, definitely true with Paketo build packs, there is a compatibility guarantee. There's a interface called the application binary interface that guarantees that computer compatibility of the OS layer to the other layers in the application, which are the ones that are added by the build packs, have to do with your application source code and any kind of runtime like a JRE, for example. So with an OCI image, the OCI image is represented by a manifest that has pointers to these various layers.

And so with a rebase operation, what happens is that you get a new image, a new manifest file, which has its own new SHA. But the pointers in that new manifest are pointing to the same application or build pack layers as the previous image. And only the pointers to the run image are updated.

So this operation happens within a few seconds on the local machine. It can happen in sub subsecond on a registry. And with something like the Pack CLI, you do this imperatively one image at a time.

And with a platform like KPack, which you is the one that gets installed as a service, configure declaratively on Kubernetes, for example, that can operate on a at scale and do an operating system patching of all of your images in a very, very quick amount of time. So I'm going to show you that as well as inspections. Now.

So from a transparency point of view, let's do the inspections first, I'm going to be using the the image that Andreas just built, which is right now in the GitLab registry for that job application. So I'm going to use a pack inspect image command. So previously I showed you the Inspect Builder Command.

Now I'm doing an inspect image and giving it the image name. Now, if I just provide the first part of this command that will give me metadata provided about the orchestration level. So which build packs were used, what versions were they?

But if I add this minus minus BOM, which stands of "Bill of materials" flag, then it gives me information provided by the individual build packs, in this case, with all the information provided by the Paketo build packs that were applied for the Java application. Now that returns json, I'm actually going to show you that in a browser because it's easier to read. Ok, so we can see, for example, information added by the Bellsoft Liberica, the JVM build pack.

We can see that the JDK and the JRE are both version 14. So that's again, if you look at this from an operational point of view, if you have an image, the ability to take that image and inspect it and know what's inside of it is incredibly powerful. So we have that kind of information.

And just as an example of a scroll to the bottom of the file, I can even see that the Custom-Build pack was applied last. So that build pack itself doesn't add too much to the metadata, but it's definitely identified. And then before that, the Spring Boot build pack was the prior one to run.

And it so happens that with a Spring Boot build back with Paketo, that actually adds all of the Java dependencies as well as their version into the metadata, the informational metadata. So I can see that Spring Boot two three two is being used in this application and every dependancy and its version. So, again, very powerful from a governance point of view and from a security perspective and from a transparency perspective for an organization.

So. Let's go back to the demo here now. Switching topics to that OS rebase.

The way that that would work is that I would you can do this on the registry itself as well. But I'm going to show you this demo running locally on my machine. So I'm pulling down the image that under just built on GitLab and then I'm going, this next docker pull is basically saying that let's say there's a vulnerability found on the image that exists, but a new operating system was released.

So this is me obtaining that new operating system for downloading that new run image. And I'm going to do the rebase locally on my machine, using the same builder as before. And that builder is looking for a base image that's tagged as base CNB.

So I'm just taking this updated, fixed image that I just downloaded and re tagging it locally so that my builder will find it. So then all I have to do, I'll show you the different images that are here. But I have this.

So this is the existing image ID and then I have the the run images. So I have the old one here, which is the one with say Heartbleed or Spectre in it that's now untagged. And the new fixed one is now tagged as base CNB.

And just for convenience, I'm going to go ahead and save the old ID of. Clear that. That's not the right company.

D. equals that. OK, so I'm going to copy that ID, which is the idea of the image that we've just pulled.

And now I'm going to run the pack rebase command. And so this on a local machine, it does take a few seconds, but as I mentioned, this is a subsecond operation on a registry and it can be run out at scale with with a platform like KPack. So incredibly powerful from a security perspective.

But this basically so it's produced a new image so that so I have a patched image. And just to show you what that actually does in the background, the Docker Inspect Command gives you the SHA of every layer, so including the filesystem layers. So they compare the old and the new, you can see that three of the layers and the root FS, these are the root FS layers have changed, but there's no other difference to the manifest file.

So that's a rebase. So now I'm going to go back here and just just to wrap up with a few summaries. So we hope that this has given you a good insight into how cloud native build packs works and what the benefits are.

So as a project, it's extremely easy to use. You saw the Pack build command. The Pack rebate's command.

Of course, GitLab Auto Devops makes it even easier, but it basically is a project that establishes a unified standard and therefore fosters a growing ecosystem of both platforms and build packs. It works for a variety of languages, any language implemented by a build pack implementation. So it really is applicable to many, many applications.

It it's very efficient and sophisticated in its caching system. In the state of transfer optimizations, it has sophisticated methods like the rebase, which is powerful for security. It encourages and enables this modularization of build functionality, and it gives you this transparency and inspection capabilities both into your builders as well as your images and the packs in particular.

As you saw, they support various frameworks. They incorporate best, best practices for each of these frameworks. They include the base images with the AVI compatibility guarantee.

They're actively updated. So you have the latest patches both in upstream for runtime as well as OS packages available. They there's been a lot of investment within the Paketo project in particular to modularize the build packs, and they're easy to configure either through the environment variables like we saw with the Java build pack or through a build YAML file.

So that's really everything that we had prepared for you. Please feel free to contact us with any questions. We're happy to help you further your knowledge into cloud native build packs.

And if you found this interesting, then hopefully you'll be interested in passing and stopping by our virtual VMware booth and hopefully joining our demo on the demo stage later on in the GitLab conference. But you can find online. So thanks a lot for joining us.

Thank you.