Tracy Ragan, DeployHub | DevOps Experience 2022

Hello everybody. I hope you're all having a fabulous devops experience. We are talking today about moving beyond devops and I'm going to talk about moving beyond devops to devsecops.

It is a topic that we've all become sort of passionate about in the last year and I'm going to kind of cover why and where we're going with it. All right. So as all of us know software's kind of hard to recall what we generally do if there's a bug is we push things forward and we always kind of move forward across the the pipeline.

Sometimes we do a rollback but generally things are pushed forward. We don't think about software in the in the way. We think about a jar of peanut butter.

So for example, we if there's something a problem with a jar of peanut butter. Pretty much the peanut butter manufacturer can tell you what region is in and who was sent to but we can't do that with software. So what happens is it teams cannot ensure that the software systems are sending out to you or that you're sending out to your end users are safe for consumption.

And a lot of this has to do with the low-level dependencies around open source, or potentially even dependencies around inner source. The problem is is that we as it teams, we cannot easily know and control the ingredients that we're using in our software. So our pipelines are missing some really critical steps when it comes to really being able to understand what we're adding what we're putting in our software and a list of ingredients for that software.

So is this a problem? Yeah, I kind of is just becoming a pretty much a massive Global problem. I don't want to I don't want to sound alarmist but it's kind of true we are we're facing it a time that there is a lot of software being written.

A lot of Open Source Code being consumed and the systems that we create are now overly complex and to be quiet honest. It's almost impossible to secure with today's Pipelines. So who am I I am Tracy Reagan.

I am the CEO of deploy Hub. I am the community director of an open source project called ortilius, which I'm going to talk about briefly at the end of this. I'm on the board of the open ssf, which is the open source security Foundation.

I'm also on the board of the CD foundations technology oversight committee. I was a founding member of the eclipse Foundation. I hope found the CD Foundation.

I'm honored to be a host of tech strong women. I'm at devops Institute Ambassador and I have about 20 years of experience doing this. I love to chat.

That is my LinkedIn address. If you want to ping me. I'm happy to chat with you about this topic or any other topic for that matter.

So devsecops is a growing segment of our industry. 16 billion by 2029. It is a growing segment of what we do.

There are many different tools that are entering the market many of them fit in well within the devops pipeline some of you may be consuming some of these tools but overall we have work to do and this depth Bob's Market. Is it going to continue to grow over the next decade? What's driving it we all know the story.

I think the the log for J's story is one of the most compelling stories of 2022. I say that because it happened in the end of 2021 and many of us started addressing this problem in 2022. So if you don't know about the log for J problem, there's a lot of Articles out there on it.

I highly recommend you take a look and read and understand what occurred. This is not necessarily a problem from because of that developer for log for Jay did something nefarious. But a vulnerability was found in log4j that allowed hackers to take control of the system where it was running.

So basically allowed you to exit out and do whatever you wanted to in the system. Which is really bad. It was a very bad vulnerability.

About 44% of Enterprises were impacted by it. Now if you think about that, that's that's a lot of Enterprises. And between December 9th and December 21st, semantics intrusion detection system.

They claimed they blocked more than 93 million log for J. Related exploitations attempts on more than 270 unique machines. Now, that's just semantics software that they reported on I we can just imagine how many systems once that that vulnerability was found.

How many systems were being we're being hacked and companies were scrambling to figure out where is log for Jay running. This is the core problem. We as devops a professionals cannot easily answer for our siso.

We can't say hey security officer. We know that log for Jay is running and these particular applications that are installed in these particular environments. So we have some work to do on our pipelines and this problem is a problem of the pipeline.

It's not something that we can pass to somebody else. Now. There's been other security tools out there in the market.

There's you know, there's pen testing there's you know threat detection. There's lots of different kinds of security software. But what I'm talking about is the core of how we build our software knowing what in the software and how to respond to a threat Part of our problem is the complexity of the software.

We're now creating. Tyler Jewell, I always quote him on these because the research he does is incredible. His insights are spot-on.

That's just to say the least and he really makes you think about what's next in terms of software development devops and security. And his uh his 2022 developer LED landscape. He's talked about this complexity that we're facing and he says that software systems have become so complex that it's given it getting too difficult for humans to reason about the system.

Now, if you've seen my talks before you've probably have seen me quote this but because and I do so over and over because the more I tell you this finally you're going to start listening I think And the complexity he talks about spans even small teams. So he says even a small teams with 20 dependencies have a broader committograph than a team of 10,000 Engineers. That's a big statement.

He also points out that securing the software supply chain has lead to the top of the CIO and CEOs imperatives for 2022, you know data officers are have been always very concerned about the data that they're managing. But cios necessarily. I don't know if they thought about it.

Now that security problem is starting to really infiltrate the entire organization cios have always been worried about the security of software. But oftentimes we get more focused on delivering features to end users and that is what becomes the priority so we discovered esphombs this year. I have always talked about esperms.

I sometimes people looked at me like I was green for talking about esperms some of you may know I have another company called open make software and in 1995, we started a process of managing the compile a link process which allowed for a build audit meaning that every single thing that the compiler link or any other resource compiler touched we reported on because it was important. We had want to know not a bill of material oftentimes in the beginning was seeing As a list of what was checked out to the build directory. It has since grown quite a bit since then but even a list that's checked out to the build directory isn't inaccurate list because your your build script can reference files from other places other than the local Bill directory and often does so for example compiler libraries.

There's maybe other things that could put out there that end up in your package and that's what happened. In some other famous breaches where in the various dll got put into a package and we had another issue. So as Farms are now becoming more and more important Jim zimlan who's the executive director of the Linux Foundation?

He says that esquams will play an essential role in building more trust and transparency and how software is created distributed and consumed throughout the supply chains and I would like to just say throughout the devops process because that's really where it has to be addressed. And this is not an easy problem to fix. We really have to put some time energy and focus into this problem and we have to have a dialogue amongst all of us to sort out the best way to solve this problem.

I'm on as I indicated. I'm on the open source security Foundation, which is a foundation under the Linux foundation and under the part of the guidance of Jim zimlin where we are working on a mobilization plan to identify areas where we as a community the it. We can start addressing this particular problem.

I have a link here software bill of materials and cybersecurity Readiness. I suggest that you Google that and you take a look at the research that the Linux Foundation did on s-bombs sbom usage sbom consumption. So you can begin understanding the challenge that we have ahead of us.

And yeah. this so prevalent that it trickled all the way up to the White House all the way up to the Oval Office and President Biden did see this as a serious problem serious enough To establish an executive order for improving the software supply chain and really what it said was. Hey, you know what guys if you're gonna send software to the government if you're a vendor of ours and you're writing software to us, you'd better know the ingredients.

We want to know when a recall is is needed. We want to know and log for J is bad and we want to know where it runs so we can fix it as soon as possible. You can just imagine what the US government went through when the log for J hack was brought forward.

I'm sure they scrambled and worked very hard like we all did to get that fixed. But the core problem was we didn't know where it was running and that is a bad thing. We didn't we all get enough for that because we did not understand what we were consuming and keep in mind.

It's not just log for Jay. It's the version of log4j and the version of your application. That's consuming it.

So it goes beyond just knowing where one particular library or packages? So, yeah, if you're if you're now submitting a software to the US government, you're going to have to produce an application Level. Ask Mom for the for the software system that you're delivering what we typically typically call an application.

So that's what they're going to they're going to ask for and I believe that started now in just just last month. They they signed it in May and I believe it wouldn't force in September. So what is in an s bomb after all how do you how do you harden cyber security with with a software bill of material report?

So there's just three things we should think about the most important one is a list of the software dependencies that the open source or the inner Source consumes. What is it consuming and of course to get really down you're going to have to go find what those those dependencies are and what they're consuming. So this can Traverse down a pretty deep tree or set a Roots.

It's kind of under the ground under the surface. That's why it's hard to find now early on S bombs were used by companies like ours deployhub that we provide software to to Enterprise companies and they wanted to know if we were using any open source software that had a license that allowed the or did not allow them to consume it. So s bombs licensing was a primary aspect of s bombs early on but they also show ownership where it came from its Providence and any of the lower level dependencies.

Like I said now with those lower level dependencies, you can start generating and deriving your vulnerabilities that information allows you to pass something to a list of dependencies for example to a cve database and get back your what Your vulnerabilities might be and if there's anything in red that you might want to react to But truly it is becoming very very complex. And I really do like to to equate it to herding cats because if you ever tried to do this if you ever looked at your dependencies, you realize how spread out they are and just calling kitty kitty kitty doesn't always work very well. They may be chasing a mouse and deciding that they're have no interest in listening to you and that's kind of what dependencies do they're just out there just kind of replicating themselves and running around and what we have to do in devops is to start documenting where they're at.

Even if we can't control them we can know them. So there are two primary s bomb standards. I'm not going to go deep into what s bombs fully include and what the standards are this I pulled this article out from a company called settle top.

I thought their article was really really well written and talked about some of the differences between spdx. Icon DX spdx is part of the Linux Foundation a cyclone is a part of a different open source Cyclone has a lot of Tools around it spdx has a lot of adoption and it I don't think it matters to be quite honest. What matters is you start generating s-bombs?

You start looking into this problem and start understanding how you add it to your devops pipeline. So while we have this really hard problem to solve in a monolith World Cloud native just made it harder. If you think about what we do when we generate a nest bomb generally s bombs are generated at the time of the build.

So if you're creating a monolithic application where you're compiling and linking all of your objects together as one binary or darfile or set of binaries, you can pretty much generate a application Level s bomb with these the tools that I referenced before. When you start doing microservices and you start decoupling your application, every time you create a container, your container build is going to include that step. Which means that all the low-level pieces that make up your logical application have their own s-bomb.

So now we have to sort out how to create a logical application s-bomb in a decoupled environment. I bring this up because while we have made some strides I think in monolith and fixing the monolith will be Challenging but doable it becomes much harder when we have a decoupled architecture. And as many of you may know who have listened to me talk before I'm a huge fan of microservices.

I believe in reuse in other words, maybe it would have been great. If log for Jay had been a microservice and when running in one container in our our clusters no way to do is replace it there and not recompile hundreds of applications. Yeah, that would have made a lot of sense.

Right and I believe we'll get there. I believe that is where the industry is going in terms of building out these packages and creating reusable components that we all can can consume and maybe when we do that, we'll have more control over them. But getting there requires that we have to think about our devops Pipeline with microservices moving across the pipeline all day long and impacting multiple applications.

And every time that happens a new s-bomb for the application Level is created or needs to be created. So this was sort of in this article this this document that was done the software build materials cybersecurity Readiness. It says the most pressing issue was identified by 62% of the overall sample was the need for industry consistence on a consensus on best practices to integrate the production and consumption of s-bombs into software development the production and consumption of s bombs occur in devops.

However, it goes on to say there's not yet a consensus somewhere s bomb production or consumption should occur in devops. That is our that is our our challenge. There are three types of vest bombs that we need to think about in our pipeline.

Source to artifact s bombs, which is your Source artifact to artifact as bombs, which is your build who's using the log for jpackage for example, and the deploy s-bomb. What is the OS level dependencies look like that may impact runtime. So you you potentially have S bombs that are going to be at multiple stages of your pipeline.

Challenges so every bill is great script needs to be updated to generate a nest bomb. If you have hundreds of build scripts, you're going to have lots of work to do to add the generation of s-bombs that means container builds and monolithic builds. This is my biggest complaint even if an s bomb is generated.

It is not consumed and it's left underneath the build directory. So in case you need it someday, you can go look at it. But there's a lot of devops intelligence in that s bomb that we should be using but there's no easy way to centralize the s bomb data to make it relevant and maybe this is why we're having difficulty deciding what to do with s-bombs.

Now Tyler Jewell makes this a suggestion on this. He says Service governance will be the creation of service catalogs metadata stores about how the software was created and intelligence systems and inform stakeholders or insights for the system. So how do we pull the devops intelligence into a single place?

And he's talks about this thing called a service governance? Which of course made us very happy because we've been working on that and if you've been watching what the ortelius project is doing or tilius is an open source project and it's part of the continuous delivery Foundation. It's a unified component catalog and it's designed to track the versions of your code along with all of the pieces It's s bombs.

It's a cve's is key value pairs any of the configuration data or the software composition data of that particular object and we stored into one place now on a decoupled environment. What we also can do is roll that data up to The Logical application Level so that you can generate a logical application SB or a logical application logical application cve and now we're starting to think about how we can build on that data to do more predictive work, but we're really talking about a centralized catalog. That takes all of the stuff that we're working on sbdx or Cyclone your cve what's going on at Sig store things by Nest or you're scanning by J frog your your deployment information that you get from Argo or Spinnaker or home your your commits that you're getting from get and pull that in to a central Intel evidence store and rolling that information up to the highest level that logical application things like provenance your dependencies your licensing that's pulling that's pushing forward and then giving you what you're available ingredients are what your impact is your exposure your application s bomb and the ability to start working on zero trust policies because we're going to have the zero trust data now we put it in the pipeline.

So we get or tilius gets triggered at any time of build is done so we can pull that s bomb information new container registry tells us there's a new container out there a new container tells us that there's a new version of an application because something Changed and then we pull the information back from the deployment to say where it's where it's at. So what the benefits to the pipeline that that changes your Dev pipeline to a devsec Ops pipeline, it reduces cost because it provides a central location for everybody to consume these objects. It exposes the risk.

Anybody can look at a dashboard and see what that where this what the s bomb looks like what the cve's are. And it restore some control because now if you have a problem there is easy to report on the vulnerabilities and it's really easy to say where is log for Jay running. So a simple search of the database will be able to tell you precisely where log4j is running who's consuming it and where where it needs to be repaired and ultimately at the end of the day, that's what we're all really working on.

There's a lot of other security pieces that go around this but pulling it into a central database and being able to act on it is what arterius is doing. So thank you. I hope you all can get started on building out a Dev SEC Ops pipeline.

I would encourage you to take a look at the artillas open source project. I've check out our GitHub become a contributor. I've call me reach out we can chat and if you have ideas about what artelia's could be gathering where we're so happy to hear any input from the community because like I said, it's time now that we all get together and start talking about what the devops intelligence looks like and how we can be part of the security answer.

Not the problem. Thank you so much, and I hope you have a great great experience at devops experience.