Gideon Wulfsohn & Jackie Meshell & Krasimir Angelov – Ship Code More Securely with GitLab and HashiCorp Vault

We have a trio to tell us about shipping code more securely with GitLab and Hashicorp Vault. In Cloud native apps, there are a lot of secrets to be kept in, carefully managed between more components of the overall system. Solid secret management is critical.

Gideon Wulfsohn is a senior solutions engineer at Hashi Corp. Jackie Meshell is a senior product manager for Ops at GitLab. Krasimir Angelov is a back and engineer for release at GitLab.

Together, they're going to share with us Hashicorp and GitLab have come together to help customers scale their secrets management. You won't want to miss this. Let's get to it.

Wow, I can't express how excited I am to deep dive into Hashicorp Vault and GitLab's authentication and secrets management strategy. We offer a really great experience for developers and operators combined. And first things first.

My name is Jackie Meshell. I spend my days at GitLab trying to make a really great experience for customers to ship their products to production as fast and as secure as possible. I'm joined by one of the best engineers I know and a speaker from Hashi Corp as well.

Hey, my name Kraz and I'm a backend engineer and GitLab. Thanks, Jacki. I'm Gideon Wulfsohn, Solutions engineer at Hashicorp.

These days it's a little tough to be a company. You must work faster than your competitors. You also have to be on the lookout for nefarious actors looking to breach, looking for breaches and invulnerability is inside your software and back office systems.

And it's compounded by the fact that in 2025 we're expected to see seventy five billion different devices connected to the Internet. And by the end of 2019, we saw that there are 66% of the enterprises making some sort of cloud transformation and over 83% of the enterprise workload will be in the cloud by the end of 2020. This is not without risk though.

And in fact the average data breach costs public companies over $160,000,000 dollars. All of this is an incredible amount of money in order to fuel the innovation that moving to the cloud offers. And there are some impressive numbers.

And in fact, we know today that software is really defining everything. There's a cultural shift and a digital transformation that you must select the right tools in order to enable your different teams and individuals to succeed across your software development lifecycle. In order for you as a company to react to your market trends faster, you must streamline and reduce waste within each of your tools so that your teammates and your diverse tools can work together all along the same thread seamlessly.

And when we really think about the automation that tools around GitLab and Hashicorp can build together, they have an incredible story and narrative. Cool. So yeah, when you think about going end to end an idea from trying to build out this pipeline, get in your automated deployment in place and really being able to operate where now you're moving from systems of record to systems of engagement.

Hashi Corp introduces four core products: Terraform, Vault, Consul, and Nomad to handle the provisioning, security, connection, and then eventually scheduling of your application on your dynamic pool of compute. So this all really plays in really well with the pipeline that you would be building out with GitLab to iterate faster upon your ideas. Adopting Vault is one step of the security journey.

When you bring Vault with the toolset like GitLab together end to end, you can adopt a modern software development platform. And in fact, using GitLab brings ideas to production faster. You can leverage our portfolio management and project management functionalities, issues and Epic's to detail the foundation of the scope that you want your developers to build and control within your source code management tool set where you can see lots of code reviews and collaboration occur in GitLab the same place for you to find your requirements and issues.

And additionally, you can save a bunch of time and reduce waste by automating your pipelines to build and test improving the speed and deployment to production with our capabilities as well. All the while, we're of an undercurrent of security and management functionality to improve the end to end functionality that you will have and your GitLab experience. The beautiful thing about GitLab and Vault and the beautiful thing about GitLab and Hashicorp in general is that both of these companies are strongholds by themselves.

But when you combine them together, you provide a really great experience for developers and operators to ship their code faster and more securely. And they're working together as if they were built with each other in mind, in a multi cloud related experience. Gideon will take a moment to talk about this solution together.

Yeah. So really, when you're thinking about, OK, we're trying to operate in an environment of heterogeneity, GitLab is a great starting point for all of your developers. They're going to be interfacing with the evolution of your software by way of commits and eventually merge merge requests that build new versions of your app throughout this.

Process, we really want to come in and introduce this concept of everything is code and especially infrastructure is code, where in now, you're actually defining your updates to your infrastructure and codified templates written in HCL, and you then go out and can modify your infrastructure without having an operator in one of the cloud dashboards clicking buttons. And you no longer need to develop this kind of PHD and identity access management for each and every cloud vendor. It's so powerful using all of the offerings that Hashi Corp has alongside a tool like GitLab.

And in fact, all of the change that we're seeing in the market and all the variety of tools that you have available to you, it's really important for you to select the right ones in order to navigate these critical times. And in fact, in order for you to avoid the pain and the cost associated with mishandling of credentials and secret information, you can implement a tool like Hash icorp Vault alongside GitLab CI CD to build out a premium secrets management strategy for all of your pipelines. Gideon will take some time to dove into the benefits of Hashicorp Vault.

Awesome. So, yeah, today we're here talking about the integration between vault and a GitLab CI CD pipeline and really trying to boil down the challenge of secret sprawl. You have access tokens that might be involved in eventual deployers.

So you need to access something from a database via an actual token versus just kind of a one off secret. These are the kinds of pieces that if they're just sitting in code repositories and there's no audit trail around access or potential lease expiry around, say, credentialled, that's abstracting away the actual token. There's a lot that can go wrong.

So we're here partnering with with GitLab to boil down this challenge to something where with two lines of config, you can really automate away a lot of the challenges that developers face today with trying to adopt a secure system for managing secrets. So all of this starts with the actual identity. Where you come in are authorized gain access to some sort of token, beginning with the initial client.

And then finally there's transfer of information the tokens used. You can carry out your full continuous integration, continuous deployment process. So then really thinking about, OK, so this is what it looks like and kind of a one off scenario, how do we actually make sure that the availability and scale is there for an organization and start thinking about the crawl, walk, run of adopting a system like this?

So with vault and really with being able to have all of your secrets in one place that you want to make sure that this system is fully available such that your developers are iterating and able to make their deployments as well as your applications, being able to interface with your secret management system in place. It's really important to, for one, implement a DR strategy. So really making sure that there is cross region replication and performance replicas, as well as making sure that you have multiple tenancies.

So basically you have different teams. You don't just want a single store of all of your secrets, but now actually being able to break up your secrets, access credentials into what we would call name spaces that various teams have access, control, and the ability to interface with. And this is really the heart of why GitLab is spending a lot of time to invest in a deeply integrated experience with Hashicorp Fault.

Vault brings an immense amount of strength and value to the industry and secrets management. And when we frame up the problem, organizations are trying to find a better way to build an application development and their infrastructure management workflows so that they're following the same security, compliance and regulations that every other system they're running is as well. When software is defining a business, it's becoming more and more critical to protect those assets, to ensure that your business is protected and performance.

When we look at the solution to that problem, we think about a comprehensive secrets management solution that builds in the automation for encryption of credentials and secrets, as well as being able to instrument guardrails and operational strongholds like a policy and Sentinel and leveraging the merger approval workflow process that GitLab offers. This beauty of tying together policy with frameworks builds in a seamless experience between your application development team, your operations team, your security team. So you see this end to end devsecops perspective, where all of these different teams and people are able to work together in the same places with Hashicorp Vault and GitLab effectively partnering together.

The Secrets Management Strategy at GitLab is all about embracing this transformation and shifting this identity left into CI CD pipelines and helping organizations implement a very strong operational and application security with governance in mind. When we embrace that, we're boiling down the essence of enabling developers enterprises to take control of security and making sure that they're implementing the best in class practices of how to handle tokens, how to handle credentials into their processes. Gone are the days of managing secrets to credentials and disparate places of writing down passwords on sticky notes and passing that around the office so that people can access to staff and in fact, the cost and the risks around data breaches are far too severe to not deal with that in a comprehensive way like Hashicorp Vault has done.

This is only a part of the story. We're really looking to simplify this experience. 9, we offered an initialization of Vault instance inside of Kubernetes cluster via a GitLab managed app.

What's really great about this feature is it becomes a push-Button experience in setting up a Hashicorp Vault instance, as a part of your CI process side of GitLab. As we begin to fill out more feature sets with the Vault integration, such as the secret syntax and the authentication mechanisms, as well as actually injecting and handling secrets between GitLab and Vault, the managed application will be enhanced to also use those exact features as well. So we're really excited about this investment and we'll help our users take advantage of that.

We're now going to take a moment to dove into the overall workflow of how one would use GitLab with Hashicorp end to end. Awesome. So, yeah, I'm really excited to think big picture here around.

OK, you're an engineering organization. You have security engineers, you have developers, you have operations people. And with anything, you have your developer coming in and filing a permit leading to a merger request, modifying your code.

Now you want in tandem to have the operator working with a infrastructure repository, making commits as well, and updating the infrastructure and making sure that of the underlying or a pool of compute is there to support the app that is going to be out running and production throughout that build process. With Terraform, you can now introduce automation policies as well, where once you're filing your merge request, you can kick off a number of automated tests just by way of pointing a repo with your sentinel code, with the test policies at your Terraform workspace. And then from there you can be checking for things like is tagging in place for cost governance?

Are we deploying a blob store that might have open access to the Internet? And just a lot of the other business specific checks you might want to run related to preventing resource deletion. On the Vault side, when you're running your deploy, you can actually interface with followed via JSON Web tokens wherein metadata related to the repository project and other information coming from GitLab can be used.

Then after a really exciting config process that we're about to introduce, to then gain secret a lease on a secret via that authentication flow and then finally being able to carry out your full continuous integration, continuous deployment delivery cycle. It's such a cool workflow. I love it.

I want to take a moment to allow Kraz to dove into the Vault integration segment that we have right here. Ok, let's take a look at what's going on in order to make your Vault secrets available to GitLab CI. It starts no different than your usual way of using Vault.

An operator needs to configure it. This means enabling security, changing, enabling, and configuring authentication methods, creating policies, assigning controls and eventually creating the secrets. Configuring the roles is an important detail because this is how we map GitLab groups project and CI to the Vault policies.

Now GitLab provides every CI job with unique JSON with web token, which is addressed to five six, encoded and signed with a private key. This JSON web token is provided to the job as defined by a variable. Now jobs that need secrets from Vault we use this JSON web token to authenticate.

On authentication request, Vault will use GitLab's public signing key to verify the json web token. The key is fetched using a command point endpoint provided during configuration. Now, once the json web token is verified, Vault will check claims against the bounded claims configuration in step one.

If there is a match, a token is created in the policies for the role we are using are attached to this token. When Vault returns this token to the CI job. We can configure tokens, lifetime and other properties when creating the role.

And the agent using the provided token the CI can access Bolt and perform any actions allowed by the policies attached. Let's take a step back and talk a bit more about the way GitLab CI authenticates with Vault . We decided to use the built in Json web token method because it's a good fit for automated workflows and there is no human intervention required.

There is also no need to share, which means storing and credentials between GitLab and Vault, unlike, for example, when you're using TOPS or other methods. And while it's extremely flexible, it's also very easy to use. 10.

Moving forward, it should be clear by now that the configuration is really important. This is where everything is bound together and there are three main components we need to configure. The auth method itself: This means enabling the method and providing it with the public signing key so that both can verify your JSON web token.

Usually this is done by specifying the so-called the so called Json webkey setting point so the Vault can fetch the key when it's when it's needed. You can also fetch the key and store it in Vault directly. The downside being that if the key changes, the conflict will have to be updated manually.

But we also need to specify the Json with tokens issues only token with matching value of the original plan will be accepted. Next, we have the policies. Vault is using policies to govern the behavior of clients and instrumental based access control.

Everything involved is part based and that means the right policies to grant better access to certain certain parts of the operation. For example, you can have a policy that grants access to the set of security required by your test environment and another policy that grants access to secrets needed for your protection environment. And we also have rules that are always specified by the CI job when attempting to authenticate.

With roles, you can group different policies together and in case of successful authentication, attach these policies to the resulting Vault token. Bounded claims are specific to the Json web token method. And in essence, these are predefined values that will be matched to the Json web tokens claims.

This way we can restrict access to specific GitLab users, specific projects, or even jobs or any specific references. We can have as many bounded claims we need and they all need to match in order for authentication to be successful. Combined with GitLab features like user roles and protected branches, we can tailor these rules to fit our specific use case.

For example, authentication only for jobs running for protected talks with names matching the pattern we use for products and releases. We can also specify some attributes for the resulting Vault token, such as Time to leave, IP address range, and number of users. To see the full list of options see Vault documentation of creating roles for the Json webtoken method.

The implementation we have at the moment is very minimal, our GitLab dos is to provide Egypt with a JSON web token that can be used to authenticate with Vault. While this is extremely flexible, one can do pretty much anything. It leaves a lot of work to the users.

They have to manually authenticate and access secret. This means they either need to keep the Vault client available to their job or do API request using something. So that's why in order to simplify this and improve the experience, we are currently working on adding the concept of secrets to GitLab CI Yaml syntax.

In their jobs, users will be able to specify all the secrets the job will need. These details, together with some details about the Vault server like URL and role will be passed to GitLab runner when requesting a job. Runner will use them and the Json web token to authenticate with Vault, read the secret, and make them available to the job.

Scretes will be stored as temporary files and the file part will be available as predefined variable, similar to the CI variables of type file we already have. On this slide, you can see the short form of the new syntax here we see to separate name database password and it's private key with their box in the fields we need. You can see they are stored in two different secret agents, one mounted from the default, but another mounted under the custom for this new syntax default to using the key value version to secret engine.

On the next slide, you can see the detailed form of the syntax where we can be explicit for all of the details, secrets engines, type and path secret path and the field we need. There were a few decisions we made along the way. First, we choose not to manage configuration and GitLab.

While it requires good knowledge of how both works, and it's not always easy to get it right. We think it's important to keep it outside GitLab and GitLab to control. This way, we keep separation of duties, make no assumptions, while still having all the flexibility and sides of the access only to what was explicitly configured.

And second, we authenticate and access both from GitLab runner. This means that only GitLab runner where the job is executed needs to have access over the network to Vault. GitLab itself does not need any connectivity with vault.

And in case you're using Private runners, you have full control over the network configuration and the whole interaction can stay in your private network. So while we are going to start with only supporting key version and to secrets engine and just reading secrets, we plan to expand this further by adding support to write and delete secrets from CI, adding support for other secret engines when, for example, key value with version one for database adding more claims to the Jason web token. We have a pretty good set of attributes at the moment, but is the adoption grows will be looking at adding anything we may have missed, but our users need.

And we may also look at getting support for other Vault integration methods like TLS or AppRole if there is a demand for that. Other product experience features that we will be looking at will be managing some of the Vault configuration issues or concerns that users have surfaced us throughout this experience. We want to make the Vault configuration experience as seamless as it is to use the JSON Web tokens.

We also want to support managing the Vault integration at scale and helping users who have high availability or multiple name spaces easily inside of GitLab. We're excited to continue to expand this integration for all of our users at the developer side and the enterprise. I'd like to wrap up this presentation with a call to action.

I would love for you all to join our First Look Program. We're looking always for a diverse set of participants to inform our product vision and user experience strategy. The best way to do that will be to join the first look program to get exclusive access to new features, product survey questionnaires and and other experiences.

You get the opportunity to share your perspective and voice, but you also get the opportunity to win some GitLab swag. This completes our presentation on the GitLab Hashicorp Vault integration. Thank you.