Tracy Ragan – Supply Chain Security in a Microservices Implementation

Hey everybody. Thank you so much for attending my session Welcome to Cloud native day 2022 super excited to be here. Thank you to the tech strong team for inviting me.

I'm going to talk today about supply chain security. I know you probably have heard a lot about it. But in particular, I'm going to talk about it in a microservices implementation.

So first thing to know is that s-bombs have finally arrived. We have started thinking about s-bombs and those are software bill of material reports if you have never heard of the term And we started hearing about it because of the Biden Administration. And as Jim zimlin from the Linux Foundation points out that he says that s-bombs are playing a central role in building more trust and transparency and how all software is created distributed and consumed throughout the supply chain.

If you haven't if you need to learn more on this and all of these should be educational experiences, right but if you need to learn more about this, my suggestion is to read this look up this article software bill of materials and cybersecurity Readiness by the Linux Foundation. It's got some of good information about the use of s bombs and how we're going to move forward. And if you didn't hear about the Biden administration's executive order, they finally realized that maybe the best thing to do for hardening cybersecurity is if you are producing software for the government, you need to be able to provide an s bomb and that s-bomb needs to be deep enough for them to understand all the open source modules that you're creating.

So again, here's another link. You can look at the White House briefing room presidential actions. Look for the executive order on improving the nation's cybersecurity and it will talk about that executive order and why that the government is looking for s-bombs.

So meet me. I am Tracy Reagan. I am the co-founder and CEO of deploy Hub deploy Hub is a governance catalog around microservices.

I'm the community director of an open source project incubating at the Olympics Foundation Under The Continuous delivery foundation called artillious alien is our logo. I've been recognized as a top Visionary and devops. I'm on the board of the open source security Foundation open source software Foundation, which is part of the Linux Foundation.

I'm also a founding board member of the eclipse foundation and the CD Foundation as well. I'm a devops Institute Ambassador, and I've got quite a bit experience in this area. So if you ever want to chat reach out to me on LinkedIn, you can find me at Tracy Dash Reagan Dash OMS So let's just think about what we are talking about when we when we talk about s-bombs and the supply chain.

So in an s in in the supply chain discussion that we're having we're really focused on open source software. And the reason why is because open source software impacts so much of what we do. It impacts internally developed code and common code some of you may have common a team that writes common code and everybody consumes that common code that common code probably consumes open source software as well as the application teams consuming other open source software.

It also has to do with tracking changes in our devops pipeline. How do we track the auditing of the pipeline itself? And just how do we govern Services across the organization?

So open source software really impacts all of these. Now the threat risk to the supply chain and an open source and why open source tends to be the focus of managing or thinking about the supply chain is first of all intellectual properties, it was the attorneys who originally said. Hey guys, if you're using consuming open source software you need to make sure it has a license that allows us to use it properly.

This was a particularly interesting for vendors because of you're selling software like deploy have does we had to make sure that we weren't including anything in our software that had a different license or license that restricted the use of that particular open source software. Core developer practices. I can tell you how many and you all know this how much how hard we work to Define good practices around delivering software around our life cycle how we you scan our software code reviews all of the work that we do internally for for our internally developed software isn't necessarily practiced in open source software and we have learned that there could be a lack of security in some modules the log for J is a perfect example of a security of breach that impacted so many so many developers and so much software across the world.

So what's really what is really the core of an asphalm a software bill of material report at the core of it. It tracks dependencies. And knowing your dependencies as the very first step in understanding your supply chain.

So I was at the open source security foundations Summit a couple weeks back in June and I met somebody and we had a discussion about peanut butter and it was interesting. He said, you know, if there is a problem with peanut butter if there's a recall with peanut butter, we have a process to notify everybody of that recall and even down to notifying somebody from a credit card transaction that they purchase a particular batch of peanut butter that had a recall, but we can't do that with software. And every we've always been able to look at the ingredients of our food and that's basically what an s-bomb is doing.

It's providing us the list of ingredients for the software that you're consuming. So if there is a recall, you know that you've been impacted. Now software developers sell them right 100% of the code all of you know that we're consuming open source packages all the time.

If you're using python as a language that alone is going to bring in open source packages. so Much of the code is pulled from open source, but we also have shared internal libraries. Like I referenced earlier that might be consuming open source software commercial libraries think about them you have purchased toolkits that you may be using they have dependencies that you need to understand.

There's their transitive dependencies. And then of course the open source libraries that that you're using they also consume open source libraries. So we are we have quite a bit of dependencies that we have to start tracking and this is the core purpose of a bill of material report.

Now the kind of the core components of an s bomb. If you've never looked at one. They're they're pretty they're pretty ugly actually, but the important stuff the supplier the provenance who created the component Providence is a big discussion around s-bombs.

What's the component name? You know what? What is it that we're looking at what the version is versions of super important because some versions may have a security problem now the versions may not.

What licenses are they using are you able to consume this and actually distribute it and the dependencies? What's the relationship of the Upstream component? And that is the core of an s-bomb?

Now what we want to know about this the the dependencies that that s bomb is reporting on is are they safe, you know, what's inside that ingredients and not all code dependencies are safe. We have to start understanding what those if they're safe or not. You can take your s-bomb and you can because it gives you a list of packages you can start understanding the vulnerabilities by passing it through a cve database.

So once you understand the common common vulnerabilities and exposures you start getting into discussion is do we want to consume this because it might have a warning or it may have something that is severe and we want to take it out of our code. So part of the part of the effort here is of the s-bomb is getting to vulnerabilities. You can't get to vulnerabilities until you know your dependencies.

Now Providence I spoke about and we one of the top things in that list is the supplier information and the critical piece of the the supply chain is knowing your provenance who created this because from there you can start building upon zero trust policies. If you know the provenance you can say do we trust this not only just trusting it but you have the cve. So now you get you've just created three levels of Defense number one, you know, you're dependencies.

Number two, you know their vulnerabilities and number three, you know who it is that you need to contact if there is a vulnerability that you need repaired. Now, this isn't something that we don't talk about a lot in s-bombs, but this is something that is kind of it's my it's my soapbox at the moment in order to really understand if they're if there is an incident from an organizational perspective, you need to know who in your organization is consuming it s bombs don't create that. This is an area that we have to start understanding and monitoring.

So in other words with log for J popped up at before we all got excited about supply chain and S bombs wouldn't have been nice if you had a single location to go to that said this log for Jay this version what applications are consuming it. It is data and that data is extremely important and this data lives underneath our CD Pipeline and what I like to call devops intelligence, which sometimes we leave it under there and we don't do anything with it. But really each team must have some clear insights into what they're using and to be able to report it higher up.

So when I talk about when I just described s bombs for the most part we think about s-bombs from a monolithic standpoint. You run your build you scan your build and that's where you get your s bomb from. So a build is as Associated or s bomb is associated to a specific build release or a build number.

And if anything changes that any dependency of that bill changes you generate a new s bomb so you have lots of s-bombs but s-bombs are literally everywhere and mostly they're in text files underneath underneath your devops pipeline. Okay. Now let's shift.

Let's just take a quick pivot here. And we're going to go down the world of cloud native because this is a cloud-native conference when we talk about Cloud native. We have to think about microservices because it is the whole reason why you're really going to Cloud native.

It's not just to put your monolith into a container and use it as a deployment. Now, that's a good way to start but it's not where you're going to end your journey in Cloud native. You are definitely going to go down the microservices road when you do that we have to think about how that impacts everything else.

So instead of creating an F-bomb for your application, you're going to create an s-bomb for your microservice and microservices are these independently independent functions that are independently deployed across your CD pipeline all day long and every time they update they're going to update application the application release. So let's think about what that looks like. So in a microservices world an application is now logical it's a logical collection of features.

And sometimes you might even hear a change from application teams and common code teams. For example to feature teams. You might have the feature team that the team that is managing the actual front end for the Hipster store or the clothing store.

But on the back end you have other features a team that's worrying about the shipping service and the credit card processing. And that is how your application is built based on these logical collections of components. So what are we doing over time as we're building a really awesome Death Star, you know Amazon and Netflix.

They are, you know, they are at the bleeding edge. And if you ever look at any of these look this stuff up, it's it's very insightful to understand how they are able to how Netflix for example is still is able to allow us all to watch stranger things at the same time. It's because they can spin the stuff up and they can bring it down as they need it.

It's because it's a microservice architecture. It's a component-driven architecture and we are all moving to component-driven architectures regardless, if it's an API microservice a sequel statement a Lambda function, whatever it might be it's component driven and a collection of components make up our logical application. Now that's a problem for generating s-bomb because now we don't have an application Level s bomb we have individual microservice as Farms.

So the loss of the of the application of microservices the store sort of like traversing the oceans without the North Star. Teams struggle to really understand an application software supply chain at this point because now they have dependencies on microservices which they may know and when they package their application, they're logical application up. They're going to package it up based on these services.

But they have other dependencies because each microservice may have another dependency as well. And then there may be open. There's always open source code to be found at every level.

So Transit dependencies and artifacts this stuff becomes more difficult when you don't do a full application build and release number. An application team still must provide these s-bombs and especially in fintech or highly regulated organizations any government. They're going to be required to deliver this kind of s bomb information and really understand their vulnerabilities understand provenance begin to understand zero trust.

So how do we start doing that? This is in essence the challenge that we're beginning to see So another tip for reading and learning Tyler Jewell brilliant, man, and he writes a Blog about twice a year does a ton of research on how things are changing and he calls it the developer-led landscape the beginning of the year. He wrote an article on service governance and he says thing, you know, I'll just read it.

It says software systems have become so complex that it's getting too difficult for humans to reason about the systems they design The humans used to be the buildmeister, they could pull all the stuff together and they knew how everything looked and sometimes they would generate an s-bomb. Sometimes they didn't but he says that even the complexity spans the team even small projects with 20 dependencies. It dependencies have a broader committer graph of 10K engineers.

And that's totally true that death star that I referred to can get pretty pretty big pretty pretty quick. Even if you have a small team. So he talks about the use of and creation of the service catalogs and service governance metadata stores about how software was created.

The intelligence systems that informed stakeholders on insights for the system. And the developer LED landscape is really interesting this last one. He did is on basically autonomous coding.

So let's think about how a governance catalog is going to add. It's really what we're going to see is a governance catalogs moving our devops pipelines to devsecops pipelines and in a cloud native world with microservice, it becomes even more critical So these catalogs register components? They track component ownership.

They track component versions. They register consumers if you're a consumer, you can say you're using a particular object. And then they begin tracking The Logical application packages with versions and the Aggregate s-bomb and cve data up to The Logical application Level.

Now in general there's kind of there's there's different kinds of catalogs that are starting to enter the market. There's catalogs that just focus on kind of the dev side of the equation where developers can register their service and it automatically creates a pipeline backstage is a great example of this. Then there's kind of the this this the SEC if we had these devsecops the gray would be Dev.

The orange would be second. The dark blue would be Ops and the orange category you have the tracking of service and aggregating the the supply chain up to a logical view of an application. And this is where you can Aggregate s-bombs and cve's and start tracking your inventory across clusters.

And then on the right side in the Blue Area you're going to manage and track your services your service level objectives. So you have to think about that microservices really do or a game changer. You're not changing.

You're not tracking slos on a application. You start tracking them on a microservice. And how does that impact all the applications that are consuming it?

Now users who want to tame these microservices. So devops teams need to have this kind of governance so they can begin tracking the versions of services across clusters in a single place. This is a drift issue when you have lots of clusters out there which everybody does and you have microservices being deployed to all of them.

How do you know if everybody's using the same version of that service if there's an issue, how do you bring them up to the the correct version? Microservice developers need a way to announce the availability of their features. They just updated the they just created a new microservice.

But how does anybody know to use it? Do you send out an email send it out on your slack Channel and certainly there's there's attributes about that that need to be communicated with it. Like hey, this is the home chart we're using.

These are the key value pairs you need to think about. And then application teams need a way to logically group their microservices together in a package now you may think about doing this in terms of a release automation tool. But when you start thinking about microservices, maybe there's we have to add some some features around those release automation tools so that we can start tracking the microservices and a single place and allowing the consumers to register how they're using them.

Now security teams and support teams really have to have this kind of a view security teams are really wanting to understand from an application Level. You know, what's what microservices are being consumed? What what is the supply chain for that particular application?

They look at things based on an application because consumers consume applications even though there are collection of services. And support teams support applications. You don't have a user calling in and saying hey your cart service your cart microservice broke.

They say I can't order anything on my hipster store. So we have to start Shifting the way we think about support teams and the information that they're provided. For example.

Okay, you're the the applications no longer working can is there a difference report that shows me what just got deployed out and what's the difference between today and yesterday because I don't know in the application team didn't know either they didn't update the credit card microservice and and that's what caused the problem and once we know it's the credit card microservice who owns it and who do we call? So I as I said in earlier, I am the community director for ortelius, which is an open source project incubating at the Linux Foundation Under The Continuous delivery foundation. And the mission of artillius is to create a centralized governance catalog for that middle tier really understanding s bombs and consumers and the law and grouping The Logical application.

So what artillas does specifically is, it takes your it Hoards. This devops intelligence that I referred to earlier. It tracks component versions doesn't matter what the component is could be a could be a database component and it tracks those components based on an application version.

So it tracks those relationships it attracts what the blast radius is. It tracks the application and component inventory across clusters. and it provides the service and application s-bombs and Aggregates that information up including cve's it also provides ownership information and it does level of tracking the inventory across endpoints so that you start tracking that deployment metadata.

Just quickly. Let's go through how it works. First of all you as a producer you register your microservice and you register a baseline microservice from there.

If you're an application team you consume that and you consume it and you create a package for your application and it's a Baseline. Now once a that microservice the shipping service for example is updated. Or tilius is going to watch the the container registry and it says oh a new update of that container just occurred.

I am going to update the version of every application. That's consuming it create a new s-bomb and a new cve. And that is how it creates the it tracks a blast radius it create tracks application Level s bombs and it tracks your application component supply chain.

Underneath the pipeline a microservice developer would check in their new their register their new container at that point a new image build has been created there is integration. Of course as you do today with Version Control and artifact repositories, and then you're going to add s-bomb and cve scanning to that. Hopefully if you haven't done that that's something you should be working on and at that point in time artilius is going to look at that is going to bring that information into its Global catalog and start tracking for each application that's dependent upon that artifact.

It's going to give a new level s-bomb version new version and that difference report and some other reports as well. So when you think about it, you have a component level of s-bomb and you have an applet which then generates an application Level s-bomb. So you have component versions and a new component version creates a new application version which in turn creates the new s-bomb and the new cve so aggregating that information as a core feature of the ortelius product.

And again, it builds relationships. It tracks how it's changing the logical application how it's changing it tells you. Oh now that we've updated the card service.

We are going to impact these three different applications. And for those those support teams, they could look at an application difference report and know without the application teams knowledge that the cart service got updated and who updated that card service when it happened and it's probably the usual suspect why the applications no longer working is because the what changed last is what caused the impact. Thank you and I want you all to get started on this journey and supply chain again.

The first place to start is to learn about s bombs read up on that information. Look at the Linux foundations and document their white paper on cyber security Readiness and S bombs. io and you can find all the information that you need.

And please follow me on LinkedIn and if you'd like to chat more you can reach me at on LinkedIn at Tracy Dash Reagan Dash OMS, and we can have a broader conversation on solving this problem and we'd love to have you join the open source community and if you've never done so this is a really good project to get started on. Thank you so much everybody and I really would like to have questions so Post away, you know if you have questions we might have a bit of time, and if not I can get back to you with those questions at a later time. Just shoot me a LinkedIn message.