Navigating the Security Maze to Protect Hundreds of Microservices in Cloud-Native Architectures at Cloud Native Now 2024

Hello, and welcome to my session on really learning to navigate the maze of security issues we have when we are working in a decoupled microservices cloud native environment. I wanna thank Techstrong for inviting me today to this cloud native event, um, a passion that I have spoken about for quite some time. So I'm super happy to be here.

So, yes, we have thousands, literally thousands of containers to secure in a decoupled architecture, which means that we find ourselves challenged to really manage the supply chain that's going into all of these containers. Uh, and this is an evolving cybersecurity threat, uh, to control what is going into these containers. You know, we used to hear just about DevOps.

Now we're hearing more about DevSecOps and certainly a lot about software supply chain security, which has a lot to do with open source. And just because we're doing microservices doesn't mean that we're not consuming a lot of open source, and in fact, using microservices may make this issue exponentially more difficult. So I am Tracy Reagan, and if you've attended one of my sessions before, thank you.

Thank you so much for supporting me. I super appreciate it. I have over 25 years of experience in this area, and I am known across the open source community as a dev, a DevOps and expert in this field.

I helped co-found Deploy Hub in a company called Open Make Software. I've been on the governing boards of the open SSF, the Continuous Delivery Foundation, and I actually was a founding member of the Eclipse Foundation. So open source is not new to me.

I have been working in it and thinking about it for quite some time. I continue to sit on the, the Technology Oversight Committee for the Continuous Delivery Foundation, where we're look working on some interesting projects, including a project that I'm involved in as a community builder called TIUs, which we'll talk a little bit about today. But first, why, you know, why microservices?

Um, are they bad? Well, actually, they're not. Microservices are an amazing way to really become highly scalable, uh, more flexible and far more efficient, uh, and in some ways better in security, believe it or not, while I just opened this saying, we might have some issues with security in some ways better for security, and I certainly wouldn't discourage anybody moving to, to this, uh, this type of a platform, moving away from monolith has its benefits.

One of those bus benefits is microservices actually have a smaller attack service, uh, than monolithic applications shown here by Bill Gates holding this massive ping pong paddle. You have a far bigger attack service in a, in a monolithic, so if somebody penetrates a monolith, they can get to more pieces of the, of the application. But microservices are designed to operate independently, which means that they're, they're built and deployed independently.

And if one service is compromised, the attacker's access can be limited to the service itself, reducing the potential damage. But they also introduce a, a complex web of security risk. For example, a single high risk vulnerability is repeated across hundreds of containers that these microservices, um, are built to, uh, into.

And yes, according to jfr, this was from Jfr, uh, their swamp up in 2023, which was just back in, um, September. So not too long ago. It takes 227 days for us to contain a single software supply chain attack.

That's a long time, and we need to, to address it, and that has to do with monolith or microservices. But remember, microservice expands this problem because we are building more smaller components that all have these vulnerabilities contained within them. So the result is a lot of frustration.

Um, I talk to teams all the time, and they do feel frustrated by this, and they feel like they're being asked to do a whole lot more to fortify the thousands of pieces of code used to create software in these modern architectures. So while we have some benefits from microservices, we have some challenges, not ones that we can't overcome. We just have to make some adjustments.

We have to evolve in the same way as the architecture has evolved, security DevSecOps has to evolve with it. So if we take a look at what we're really dealing with, it's the fragmented data. So in monolith, when we ran a, uh, when we created a monolithic application, we could all, we could generate an SBU for the entire application.

It was all contained APIs, microservices, they were all built and, and, and statically linked into a nice little ball, stuck it in a container, and, and, and there we went. But microservices and cloud and, and honest cloud native architecture, and I say honest meaning decoupled, uh, it's fragments, the security insights across all of these containers. So for example, if you have a, an application that's consuming a hundred microservices, you're gonna have a hundred SBOs, you're gonna have CVEs across all 100 of these containers.

And there may be the same CVE as in across all, probably all the same CVU across all containers, versions, and inventory is, is is all fragmented. So that data becomes not centralized anymore. And because of that, it adds the com that complexity.

Now, you know, some of you may be using spreadsheets. There may be many ways to, to manage this information, but the fact is, we're having to deal with more pieces and more complexity when we deploy and manage a cloud native architecture application, a kuber Kubernetes microservice application. So, again, the problem becomes, if there is a vulnerability across microservices, every single container must be be rebuilt and redeployed to contain that high risk vulnerability.

So this is gonna involve more people and more effort. So while microservices are really great and they can reduce the attack surface surface, they add complexity because we have to manage more objects. And that's really the, the core of it.

And that managing more objects simply means the data is fragmented, it's fragmented across multiple places, and it also is fragmented from a perspective of continuous integration. And when I say that, I mean that when we want to add security to our pipelines in monolith, we could add an SBO M generation. We could, we could add security tooling to our one single monolithic CICD pipeline, and we would be done not so much in microservices, right?

Because every one of them have their own CICD pipeline. So now, instead of managing a pipeline for one monolith and maybe different versions of that monolith, we are managing hundreds of pipelines and maybe more hundreds more for managing for different versions of that microservice. So our applications have become smaller, more efficient, but more complex.

So you might ask, well, is it really that compelling of a problem? Let's kind of review where we're at from a community and what we're addressing. 64% of enterprises reported an impact from a software supply chain attack in 2022.

That's according to to, to Encore, 742% growth rate in, um, software supply chain attacks between 2019 and 2021. According to sonotype, that's staggering. I don't know what it is now.

We'll see when they've, they've produced their, uh, their, their next report. But that's still a staggering number, a hundred percent increase in CVEs. I think I saw, uh, read somewhere where they're predicting to probably what we'll probably have about 33,000, um, vulnerabilities reported this year.

It's a lot. It's growing every year. And 60% of organizations, by 2026, at least 60% of organizations mission critical software will mandate software bill of material disclosure in their license and support agreements, which means that we better get on it, because that's not saying that they want an SOM for every single microservice that that application consumes, it's asking for an SOM for the application.

Now, again, we wouldn't have to deal with this in monolith. This is a new problem that we've created with microservices, and it's a challenge for the DevSecOps community or even DevOps community to start pivoting and disrupting the way we do work in order to support a highly decoupled architecture that is the promise of Cloud native and MI and Kubernetes. So let's just talk about some key issues for IT teams, and we'll go through each one in detail.

But let's just talk about what we're, what we're talking about. First of all, we have to maintain a, a security compliance level for every single application. And if that application's a microservice, instead of doing one for a monolith, we're doing one for multiple microservices.

Oh, but by the way, we also need to, we need the application security posture for the application that the microservices create as a collection. So how do we do that? How do we start generating SBOs for an application that is simply a logical collection of components and independently built containers, all which have their own SBOs and CVEs?

How do we know where o uh, vulnerable, uh, open source package is deployed across all our deployed containers? And of course, our containers are deployed across multiple clusters. So as A SRE, how do we start tracking that kind of data?

How do we know where this, the, the impact is? How do we know the blast radius of a single open source package? So something that we need to be able to report on and we need to be able to use to remediate, and how do we implement standard security tasks across hundreds of microservice workflows?

Even in the monolithic world, we have millions of workflows, uh, CloudBees reported in this year's, uh, uh, state of CloudBees report, state of the, uh, the CICD pipeline that they manage 90 million workflows a month in their supported SaaS version of Jenkins, 90 million just for Jenkins that CloudBees has to deal with that is not counting all the, the GitHub actions. It's not counting CircleCI or any of the other CICD tools like teon or shipwright or Screwdriver. It's not counting any of that.

So we literally have millions of workflows that we are going to have to standardize on security tooling if we wanna address this problem. And that's generally in monolith. We're creating, you know, backstage generates literally hundreds of workflows every time you wanna create a new container.

So we are, we are building our, or we're putting our, putting ourselves in a box here with the way that we're managing these pipelines in this kind of scripted methodology. 'cause it makes it hard to update them. And then again, I referred to it, how do we deliver in a higher level reasonable application level software bill of material report, especially if we're in highly regulated industries or in industries that are serving the US government.

Not everybody may have to deal with this, but I'm guessing that most companies will want to deal with it because it gives 'em a better view of their vulnerabilities and, and how they're managing them. So let's dive down into each one. We'll get into the kind of the nitty gritty.

So managing security compliance levels. Um, don't know how many of you have been watching security, but there is a new kind of industry that, uh, or group I guess you'd call it that Gartner and some of the analysts are talking about. And that's application security, posture management, A SPM.

And that's the, that really is the ability to look at a particular component and understand how secure it is. What is its po what is its security posture? So developers, SREs in particular, DevSecOps teams or just dev teams, uh, or DevOps teams are starting to be asked to provide more details about the software they deliver.

What is the complete solution scorecard level Open SSF has something called the open SSF scorecard that really evaluates the how secure the repo is for that particular, uh, artifact. Did all the consumed services comply with salsa, which is a build level standards? Is there an sbo m Is there a way to even track vulnerabilities past doing static code analysis?

So from the, the most part, I believe that companies like jfr and and sonotype have done really good messaging around the importance of doing static code analysis. Where you can see vulnerabilities, you can see problems in your, in, in your, the structure of your code. But now we have to start going beyond a single point in time.

In looking at the application posture. We have to go beyond the con the creation of the container beyond the build and beyond the build is a world where we ha we have new zero day vulnerabilities show up after we have com created the container. How do we see that?

How do we know that what we just delivered 10 minutes ago now has a high, uh, a high risk vulnerability? Wow, that's not something we might need to know every day. If something comes up like a, another log for J, we really need to know that.

And guess what? We are being hammered on harder and harder every single day. The better we get at protecting ourselves, the better hackers get at figuring out ways around the, our, our security guards.

So in a microservice architecture, the secur, because the security intelligence is fragmented across all these, uh, deployed containers, the IT teams need the security data aggregated up to these logical domain levels. Now, maybe we need to start seeing it beyond just one container. Maybe we have to see our, our app, our security posture at a logical application level.

So an application team who's consuming con microservices can understand their application posture because one microservice could bring down their credit rating. It's kind of like keeping your credit score high. Um, the other thing is we might be, need, need to be looking at a, not just an application security posture, but an environment security posture.

Let's look at it from an environment point of view and maybe at a director or manager level of point of view. So there are different domains or context in which we can see application security and the posture of application security beyond just the container. And if we start tracking that information, we may have better insights into how, how vulnerable our environments are our applications are, or how, how, how a particular team may be consuming open source that makes them have a lower, uh, credit score than a team that might be doing better work.

These are views that managers need to have, and directors need to have to understand areas of problems. When we don't have this, the, the evidence, when the evidence is so fragmented and we don't have a good view of these higher level domains, tracking security just becomes more difficult. Where is log four J running?

I use this even though log four J, we've heard about it over and over and over, but not everybody's still protecting themselves from this type of, of vulnerability. We need to be able to answer that question. Are we using that version of Log four J and is it running right now in any of our production environments?

And how problematic is that? So again, at build time, your static code analysis will tell you what OS packages were used. But what about beyond the build?

Because vulnerabilities are found every day, we need to start tracking the vulnerability to the deployed environment. That intelligence is what we need to expose, um, the risk level and to determine how quickly we need to respond. So if you were not running a version of Log four J that that had the exposure, you didn't need to worry about it.

And one report should be able to show you that. So connecting the package, um, the vulnerability, the package, the package to the container and the container to the deployed endpoint is part of that problem. It is what we like to call understanding your blast radius.

If there's a vulnerability in a, in a OS package, what is the blast radius of that vulnerability? And what are my runtime environments that are impacted Easy onboarding of security tools In CIC I'm gonna talk a little bit about this 'cause it is a, something I am super passionate about. We need to solve this problem.

If we have millions of workflows. We need to make it easy for teams to add security tooling. Every microservice has its own CICD workflow.

They're independently built and deployed. Every CICD workflow must be updated with security tooling. It's a manual huge task.

This is an area that we need new tooling and new new thought leadership to disrupt the way we have been managing our workflows for quite some time. Um, even the ci, even the Jenkins community has spoken about this for quite a long time now. The use of plugins have slowed down our ability to be agile ourselves, even though we, we constantly, uh, preach the importance of agile development.

And then number four, this logical a application. sbo MI think if you start looking at SBOs and start reading and learning more about software bill of materials reports, you'll start seeing that there's a lot of discussion around SBO m sharing. And that's because when you have the need to, to be able to report higher up and you have an SBO M report that you're gonna send off to somebody that says what your SBO M is for your application, that application has a lot of lower dependencies who also have SBOs and that information has to be aggregated up and probably de uh, normalized so that you don't have all this repeated data.

So I, I believe you're gonna start hearing more about teams asking for these SBOs and sharing SBOs and how to normalize the data so it's not so hard to look at. So the question we should be asking are, are there any op open source solutions that can help? And that's the good news.

I'm going to introduce you to what we call Hero Project. The hero project is being, um, uh, initiated by the Continuous Delivery Foundation under the CD events project and the cd. If you haven't learned about CD events, we'll talk about it in a minute, but the whole idea is how do you standardize a CI ICD workflow with added security tooling or any tooling that you wanna add to it without having to always update the workflow files?

We have plenty of workflow files. We've gotta stop touching them, let them run. Let's figure out another way to do this.

Um, Orillia is part of that. Screwdriver, Ker Jenkins, Teton and shipwright are all part of the CD foundation. And the Harold Project is hoping that those, those CICD tools will start looking at ways to solve this problem as a CD foundation initiative.

So what the hero project is, is taking existing pipelines from whatever CICD tool you may be using, and create a proper message queue and listener to broadcast events on certain, uh, on, on, on certain events. So for example, a published event may broadcast, uh, the e the execution and Initiate SBOs for just for example. And that would be a really good start as a POC, is that we could get that done.

But the second thing is, is, you know, why are we generating this kind of data if we're not consuming it? So a central evidence store becomes important. And if you wanna learn more about central evidence stores fidelity, uh, end user for the CD Foundation published a document that they've been working on that does use CD events.

They've customized it somewhat and it includes an evidence door to start gathering this information. So two things we can, we, we can do is we can with TIUs and CD events and the existing, uh, projects, we can look at standardizing the workflows and collecting the evidence so we can start sharing this information and answering questions that we asked in our use cases. So what is CD events standardized workflows that's simplifying it 'cause it's much more, um, interesting than just that, but cd it's a co right now it's a common specification for, for continuous delivery events enabling interoperability in a, in a, in the complete software production ecosystem.

I like to say it frees the pipeline from plugins to open up fast pathways to add security tooling into DevOps pipelines. But it also opens up path fast pathways to change the tooling that you currently have. dev.

And then what is Tels? Ortel, uh, is a security data gathering and evidence store. So as each one of these microservices, all these components that we're creating for these cloud native applications, and I add AI agents to this screen because we have to start thinking about AI agents from a DevOps perspective, tels gathers that information so it's centralized.

So we can answer the question, where's log four J running? So we can show data based on a logical application or a environment or based on a, a director or manager level. io, check out what uh, we're doing there.

So at the end of the day, we are, we, the results is a heroic good start, which is why we call it the hero project. What we can deliver potentially standardizing security workflows that could get initiated at the publish event application level SBOs or just domain specific SBOs continuous vulnerability reporting, not just static when you do the build, but continuously reporting on your vulnerabilities, looking at your vulnerabilities every five or 10 minutes and maybe casting to the team that a high risk one has shown up. Microservice compliance posture and as well as application security compliance posture, central dashboard of application postures is gonna help teams understand when they're consuming microservices, what that has done to their posture open SSF scorecard evaluation.

So we can see what's happening at the, at the repo level, uh, GitHub, repo Insights, sonar Cube and Veracode, uh, Veracode results all in one place. And certainly open source inventory across all your deployed endpoints. That is what we're working towards.

So now they ask, get involved in the hero project. We need implementers, we need people to help with POCs, we need end users who wanna talk about this and help solve this problem. Check out CD events.

You can find CD events, um, their, uh, Google group to join their mailing list or go out to their GitHub and check out the specs. You can also join the TIUs project. io, check out our Google groups so you can get notifications of meetings so you can get in the conversation or see what we're doing, uh, in terms of, uh, development and how you might be, uh, able to get involved in building this amazing new, uh, way of managing workflows, gathering data, and solving the software supply chain issues that we're facing today.

And thank you. Please find me at my LinkedIn. com.

There is an about page about me that has a way to get in touch with me. Um, and you can also to follow Deploy Hub, my company on on on X at Deploy Hub, uh, proj. Many thanks again to Strong and I hope you all enjoy this session and all the awesome sessions that are offered today.