Myths and Realities of Software Bill of Materials | DevOps Connect: DevSecOps 2023

Hey everyone. Um, gosh, last time I was here I got Covid and it was 2020. Um, so definitely mask up on the planes.

Um, I'm DJ Shalene. I am a distinguished security architect with, uh, a group called the paranoids. And, uh, we've been around since 1998.

I've only joined seven months ago. We're the security team for Yahoo. So you're male, your Yahoo Sports Finance, uh, finance, fantasy football.

We help make sure that the bad guys don't get in there and, uh, don't, don't take anything from you. I'm also a, uh, podcaster. I have a new podcast called daba.

Uh, it's, uh, out on the sourced radio network you can get wherever you, uh, get your podcast from. Um, and I'm also a contributor on 5 0 5, uh, with, uh, some of the other great folks Trace Bannon. Um, and, uh, gosh, the, the, uh, it, it's, it's endless.

Um, so yeah, this presentation is, uh, myths and realities of software, bills of materials. Um, show of hands, who thought the S and SBO stood for security? That was my little joke of the day.

Anyway, it's not, um, but Deadpool makes his appearance. And slide number one this time. Uh, little story.

Um, anybody remember Heart bleed? Yeah, it sucked. Um, it was bad.

So it was back in 2014. This is the, the genesis of my journey of inventory and talking about software bill of materials, and just finding out where things are. So I was at a large healthcare company, fortune five Healthcare Company, and I was a security architect over there.

And it was a, it was a rough, rough couple of days, uh, trying to figure out where all of the open SSL versions were and our infrastructure. So what we did was we sent out a spreadsheet, uh, to everybody. So nice little Excel spreadsheet.

And of course it came back and everyone was like, yeah, we don't, we don't use it. So we knew everybody was sort of, uh, full of crap, but that's when I realized that we have a really big problem. Um, we don't know what's in our software.

And this could be libraries, it could be open source, could be fonts, it could be images, it could be anything. We don't know what's in there. We also don't know where it came from.

So did it come from Stack Exchange? Did somebody copy it? Did somebody pull it down from GitHub?

Or, you know, if it's a container, was it qua io or was it Docker hub? Like we have no idea of the providence of where our components or where our softwares come from. And then are there any vulnerabilities in it?

Um, you know, this is probably a big one. This is where inventory meets security, the where is it, and then how bad is it? Uh, and then also, you know, Tracy saying ai, that's like, where did it come from?

Did it come from an generative AI tool, like co-pilot or Code Whisper or something like that. So here's a first myth. There's really no problem here.

Um, we've heard from a lot of people today that there's obviously some issues with supply chain. And, uh, when I was at sonotype, we did a state of the software security, um, state of the op, our supply chain, we did the DevSecOps surveys, all these kind of statistical reports that came out. This one is a recent one from Anchor, and they surveyed, let me see it probably about, yeah, 2,400 projects, which doesn't sound like a lot, right?

Um, but 97% of these things, uh, used open source software. So that means 3% of people are lying on this survey because everybody pretty much uses open source components. We had 81% say they contained one vulnerability in their source code, who has one vulnerability, right?

Maybe, um, you know, it's in the hundreds of thousands of repositories we have at Yahoo. So think about how many vulnerabilities we have and what kind of volume we have to deal with. 88% of these utilize components were not the latest version.

And this has been a problem for decades, and something that I've been talking about for years is developers pin versions because they want stability in their products. Why not just use latest, right? And everyone's like, well, there's security vulnerabilities and latest, well, five versions behind.

The developers have the same view of the code and the software that you do, and they've had more time to deal with looking at vulnerabilities while you're too busy pinning your versions. And, uh, 53% of these have license conflicts. So license conflicts, this is, uh, using agpl or GPL LGPL licenses.

So you put this code in your code, and based on the license in there, the, the TLDR on this is you should be open sourcing your intellectual property, which, uh, you know, with the AI conversation, it can be injecting LGPL code in there for you. You have no idea. Uh, and then 78% of all softwares composed are composed of open source components, right?

So 22% of the software that we build, um, is written by us, right? It's not written by, you know, our developers. It's, it's written by the community.

And with that, um, that's a low number. Some web design or some web applications, service applications using React or other frameworks, that's as high as 97%. And that's one of the statistics we found at Sonotype back years ago.

So star bomb to the rescue, right? Um, I love Wolverine. He never dies.

Sort of just like Deadpool. Um, yesterday I was in the hotel lobby and was having a drink before we went out for the speaker dinner. And, uh, this guy started talking to me and he was talking about firmware.

And firmware is a big, it's a big nebulous area for me around build of materials, right? Because it's hardware, it's software, it's firmware. And he said that they're generating F bombs.

And I was like, I, I don't know about the FBO thing, but sbo, yes. Um, we got SaaS bombs, eye bombs, H bombs, dab bombs. We have every single kind of bombs that you can possibly imagine.

And, um, I think it's gonna be trendy. I'm really looking forward to looking at or going to the expo tomorrow and seeing how many people have SBOs on their boots. I think it's gonna be pretty interesting.

Um, I know that's seven years ago when I came here and spoke. I was talking about Dev SecOps. Everyone thought I was crazy, um, said it wasn't a word.

And then the next year in 2018 walked around the floor and everyone's calling it SecOps dev and, you know, just getting it completely wrong. So Twinkies and Beans. Um, last week, an episode from the Bomb went live and it featured Alan Friedman, Dr.

Allen Friedman from cisa. And, uh, he pulls out a bag of Twinkies and he holds it up to the screen. He is like, this is what an S bomb is like.

And then he turns it around and he shows us the ingredient or me the ingredients. And I used the Can of beans analogy, and I've always talked about can of beans, like, you know, it's something that's in your cupboard. And of course, you know, you got the, uh, ingredient list.

This one's fortified with log for J. Um, and if you look at the ingredient list on a Twinkie, there are so many preservatives in this thing, it's probably not the best for you. And, um, chances are your mother is not gonna recommend you eat that.

They might think you should eat the beans, you know, that might be certified gluten-free U S D A organic. What this does is it tells you what's inside the products you're purchasing. Um, and, uh, also tells you something like this, right?

It's one thing to know what's in your software, but like some things can kill you, right? May contain shellfish, dairy, wheat, not sea, whatever. Um, this is the important thing about what these labels do, and that's what software bill of materials are, is they're a label on your software.

They're a label on your infrastructure, the label on whatever you're trying to sell, especially to the federal government. And they've been around for years. Uh, this is a, I think a 1976 Jeep Cherokee truck.

Um, automobile industry's been doing this for years. It's not a big surprise. You know, you buy something in your car, your car has, uh, the whole supply chain behind it.

Um, but we also have to remember that there's things that touch your, your car, which is robots. And we'll talk about that in a little bit. So when it comes to code, um, I wrote an application last summer, uh, called Bomber, and it scans SBOs for vulnerabilities.

Um, and the reason why I did it is back, I think it was 2021, uh, I went to UnitedHealthcare and we started asking for SBOs from our vendors, and they looked at us, they crossed their eyes and they gave us a requirement, stop text file for Python. Said, there you go. And, uh, I, I looked at it, I'm like, what do we do with this?

Um, cuz we wanted it in a standardized format, but bomber, uh, came out of the idea that we would get these s bombs from our vendors. And we're like, what did we do with them? Well, we wanna scan 'em for vulnerabilities.

There's nothing that takes an SBO from an external vendor at the time and scanned it and gave you an output and said, this is all the vulnerabilities and this bill of materials that we're getting from our vendor. So this is Bomber's actual code, um, or other dependencies in the, uh, uh, what, what this application I wrote was composed of. So there's 12 direct references in there.

So these are things I actually typed in said, using this import. This, um, 16 indirect references came in as soon as I said, uh, go mod, tidy, um, and go get. And then the big thing is there's 511 transitive references.

So we're upwards of, you know, getting close to 600 things that I didn't write, but I pulled into a CLI that scans for vulnerabilities. Um, the nice thing is they scan bomber with itself, and I don't get vulnerabilities, but maybe tomorrow I will. So here's another myth.

S bombs are only good for open source software. Um, no, so bomber doesn't just contain open source components. It also contains files.

It contains 19, or sorry, 14 images, nine markdown files, one go mod file, one go sum file, 40 source files, and then 11 text files. Um, combine that with the other dependencies. And this is a little small application, like imagine if it was Microsoft Word.

Um, I don't think you'd ever get an SBO from Microsoft, but if you did, it's probably gonna be a lot more than this. Um, so you're looking at things like supplier maintainer. So that's what an SBO should contain, right?

Like who did it, um, who made it, what version it is, what the components are inside of it. Um, we can also have vulnerability information in there. So there's, uh, the ability to expand on this and SBOs are not sca.

I've heard this so many times. Like I have sca, I can like find out the exact same information. SCA doesn't give you an inventory of everything other than just components.

That's it. It doesn't give you the files. And you know, there could be, uh, a font in there with the virus or some sort of, um, you know, vulnerability that you can exploit.

You never know what it is, but it's also, these are the files that are in there and you can tell if things are changed. So, um, some of these file formats that come into, you know, cyclone DX or spdx, some of the formats out there, they have a signature on the file so you can tell if something's changed, um, which is really good from a security perspective. Cause if something shouldn't change cuz it's in maintenance mode, then you can potentially tell that something's happening that's not supposed to.

So 2021, the government started paying attention, um, came out with executive order 14 0 28, uh, which basically, you know, the TLDR on this one is, uh, if you're gonna sell to the US government, you have to give 'em an S bomb. You have to no, if the request one, yes. So there's a bunch of ambiguities inside of this executive order.

Um, and again, they just want the nutrition facts. So we have, uh, the enhancing software supply chain security section in here. It's a lot of words.

Um, put this into chat G P T and summarize it. It probably would be a lot easier to read. But, uh, uh, the, the main parts of this is that if they ask for a software bill material because they want to purchase something, you're gonna have to give it to them.

The FDA is mandating that they get software bill materials for all the devices that they get. Um, we start looking at, uh, you know, 60 days of the data of this order. The ntia had to come up with minimum requirements for a software bill of materials.

And this is what they are. So a supplier name. So for bomber for example, it's a DK FM or DevOps kung fu mafia product.

So that's a supplier name. Uh, the component name is Bomber. That's what it is.

Version of or version of the component. And, uh, unique identifiers, dependency relationships, all of these are actually it's component names, sorry, is the third party components that are in there. Um, the version, so, you know, version two, log for J version, right?

Um, dependency relationship is not necessarily mandated. So software bill of materials in the minimum specifications that were published doesn't include transitive dependencies. It only includes those direct dependencies that you have in your software.

Author of the SBO data. So this is bomber. And then timestamp is when the software bill of materials is generated.

These are some of the formats that, uh, you can find. Software bill of materials in Cyclone dx, uh, OS project came around 2017 swd, which is, uh, a crazy tag format. Um, if you see that you're probably dealing with, uh, automobiles or some sort of hardware manufacturing.

Um, sift is a new one. Uh, it's mainly used for containers, um, in sift format. So there's, sift the product and then sift the format.

And we'll talk a little bit about that in a minute. And then there's s spdx from the Linux Foundation that came out in 2010. Um, and, uh, these are the four formats you're gonna see.

Again, as we, uh, heard from Chris Hughes earlier this morning, cyclone and Spdx seem to be the, uh, forerunners in this new browser wars. Uh, but there is a, um, a difference in a lot of these, uh, SBO formats. So here's another myth.

We have all these formats. One can be converted to another. Yeah, they, they can, um, you can get any tool to convert things back and forth.

Generating SBOs is a commodity. You can use sift, you can use your, uh, SCA tools and you can generate SBO just with the SCA information in there. Um, the problem is that not every generator of SBOs generates things equally.

So if you look at a sift SBO m for example, sift is really good for containerized SBOs. It'll generate an sbam, it'll tell you what components are in a specific layer, which is really nice, especially if you're trying to figure out where something happens. Um, you translate that into a Cyclone DX format, you're gonna lose that cuz there's no layer information inside a Cyclone dx.

You can put it into some extended fields. And the same with, um, uh, SPDX as well. Spdx has the ability to be extensive, extensible, so you can put it into a field, but just because you put it into a field doesn't mean it's gonna be picked up by another format when you convert 'em back and forth.

So it's a little bit lossy. It's like, uh, taking a gif and converting it to a Jed. So here's another myth.

It's, it's too early to even think about SBOs. Um, has anybody started generating them in their ci cd pipelines? Tracy's nodding.

We've got a couple people. Well, it really depends, like if you want to or not. Um, at Yahoo, we're not under any regulatory compliance to actually generate them, um, but to consume 'em from our vendors, absolutely.

Like we have, again, hundreds of thousands of repositories. Um, we do generate them. And, uh, I do this one, this one's crazy.

And by the way, if you see my DevOps reference architecture, I'm extremely sorry for that. I've read a book called Essentialism by Greg McEwen, and it's about the relentless pursuit of less. So this is a little bit simpler.

Um, but what it does is, uh, there's four different sections of this in the, in the top right hand corner or left hand corner. Um, we have a developer who creates a container or who's coding up a container and an image. And then on the top there's a developer who's coding code an application.

So there's two things that you can do. And if, um, if you check out the bomber repository on GitHub, uh, I have this set up to be pretty much automated. Um, this first little piece, so I use an application, we wrote a DK FM called Hooks and Hooks, uh, integrates with pre-commit hooks and git.

So before you commit to the remote repository, I build a software bill of materials using sift, and I add it into the commit, and then I commit it and push it up to the remote repository. So what I'm doing at that point is saying, here's a snapshot of my bill of materials in readable exportable format for the code that I just pushed up. Then there's the release process.

So this is using GitHub actions at dk fm. And when we tag a release, we push that tag and then all of a sudden the automation starts going. So we're building it.

Um, we do release verification, we do scanning, security scanning on it. Um, we, uh, gosh, generate an SBO for it. That gets added into the release artifact.

So in our releases page, you'll see all the binaries that are built, and then you'll see an SBO that's sitting right there. The other thing that we'll do with it is bomber creates a VDR or a vulnerability disclosure report. Um, and that's a subsection of Cyclone DX format.

There's a bunch of different formats. There's open vex, there's vex, the vulnerability node inside of Cyclone dx. There's also, uh, N T I A and csa.

Those organizations are coming out with their own formats for, for VEX and vulnerability disclosure. It's really a hot mess, um, to be honest. But bomber will actually generate a VDR format and Cyclone dx, um, uh, so that you can basically take it and enrich a b a bomb itself.

Um, and then you can add contextual information, which is, which is a VEX file. So if you think about, um, when you hear vex, VEX has contextual information that instantly says it's manual. So if you're getting a vulnerability report, that's pretty automated, right?

Scan something, I get some code back. You know, it says cve, X, Y Z severity. I have to actually go through things manually from a security perspective to say, oh, this isn't reachable.

Well, we have a compensating control or we're not gonna fix this. Um, and that's where VEX gets a little hairy. So at that point, all of this information here, so the, the, uh, sbam that gets created in the build process and tagged in the release goes into a centralized repository.

We're using guac, um, and this is what we're using at Yahoo. Um, the VDR needs to go into guac too. Uh, then we also have a VD RSS SPDX file that, um, not sorry, spdx, a sift file that comes out of the container build that has the application in there.

So there's bomb number three, and then there's another bomb that goes into guac with that release. So all these four get packaged together. Um, and that's for the things that touch it.

So the assembly line of a car, if you think about the robots that touch the car, the paint dipping machine, all these things that touch the product along the way, there's an inventory for that as well, right? Um, think of solar winds, right? You know, you get, uh, uh, some injection coming in because of, uh, Jenkins, um, or think about if you had a compiler using gcc.

Um, it's really easy to, uh, customize the compiler to generate by code that you're never gonna detect. That puts a back door into your system. Um, we've seen it and it's, it's sort of scary.

So this is great. We're throwing all this stuff in. What do we do with it?

It's a big thing of software bill materials is, you know, at Yahoo, we got all these, you know, repositories, we got all these bombs coming in. Uh, why don't we generate all of this volume, um, just to look at, right? Because we don't have to give it to anybody yet.

You know, that might change with some of the, uh, cybersecurity strategy. But what we're planning on doing is, uh, getting angry evil monkey here to detect a new CVE from a feed, right? So CVEs come every day.

What we do with the CVE is we have an automated process that goes through checks to see if we have an artifact or an SBU inside of our Jira issues. Uh, SBU is how we calculate severity. We use EPSS scores to automate the probability so that we could order them.

5, but it might be easily exploitable. And that's what we get from E P S S. So we can say, do we have that inside of our system already?

If not, we create a ticket for it, we automatically score it and then we go to guac and say, where the heck is this thing? And we attached that report artifact back to the Jira ticket. So now all of a sudden we've increased our response time to vulnerabilities.

That could be a zero day, right? Logged for J two weeks of misery, you know, trying to figure out where that thing was. Um, confluence, um, before I went to Yahoo, I got hit with Confluence, which is, uh, public confluence got it taken down and uh, somebody got a lot of Bitcoin from that.

So this is the things that we can do with it. And then in the bottom right corner, there's a little third party risk governance place and, uh, vendor third party review. And we'll talk about that here right now.

So another myth, only the government cares about software bill of materials. No, you should, right? You buy something, get a software bill of materials.

Um, any kind of software product that you use get one, right? It doesn't hurt to ask. Chances are you're gonna get crossed eyes.

People aren't gonna understand what you're talking about. And I think that's gonna change over time. Um, it's gonna be more commonplace to have that as parties part of your third party risk governance.

Um, if you request them as part of your security review, there's a bunch of things that we can infer from SBOs. We can look at them, we can scan 'em for vulnerabilities. And again, that's a snapshot in time.

So you're only as good as the last vulnerability scan. But we can also look at the software versions that they're using inside their components. And we can tell if, uh, there's any technical debt, right?

Let's say they're five major versions behind, we can say, Hey, you know what? If they get popped to something and especially around this component, um, they're not gonna be able to up update this and get a release out, you know, for three weeks. Um, so that helps us make informed decision of where if, whether we're gonna buy that vendor software or not.

And then there's a couple issues around this too. I find a vulnerability in this, right? I got a bill of materials from our vendor and they dealt with this, um, when I was in the healthcare industry before Yahoo got an sbo, critical vulnerabilities in there, okay, hey vendor, you got a critical vulnerability, okay, we're not gonna fix that.

You know, there's these SLAs that we need to negotiate, so it could be part of our contractual obligations to figure this out. So you can do a couple things too. Dependency track from O osp, continuous sbo M integration and analysis.

Really pretty, it's good ui, especially for non-technical folks. Bombers scans everything. It'll scan a container too.

Um, it'll scan whatever gripe from anchor. That's what we use for container scanning. Um, and again, we just put it into our pipeline.

Um, our developers didn't even notice we put it in there, which was great. We're sneaky paranoids. So this is the second last slide here.

Myth. Everything in this presentation is everything you need to know. And as there's no EVA Monkey shows up again, um, you're only as good as your last scan.

And there's a freshness problem with SBOs, right? Every time you build software, like, man, I think we got hundreds, gosh, not even hundreds, sorry, uh, hundreds of bills from like one team a day. Um, yeah, you're starting to deal with, uh, tens of thousands of continuous build, tens of thousands of SBOs created.

There's a retention issue. If it's your vendor, are they gonna give you one every time they do a build? Probably not.

Maybe a major release, minor release. So there's a freshness issue there. Um, and then, you know, VDR and vex are only as good as your last vulnerability scan.

Once again, VEX requires context. So if somebody asks you for a vex, you could probably laugh right now because you're not gonna be able to do it. It's, um, it's all machine based pretty much, uh, for generation.

And then the last one is there's still a lot of dust around. Um, the, if you participate in some of the SBO m working groups, there's actually a SBO meetup tonight. Um, I don't have the url, but if you come grab me after I can tell you about it, um, Dr.

Allen Friedman will be there. Um, but just be patient with it, it's gonna happen. Um, and again, these things are really large, so, uh, make sure you have lots of storage and you have retention policies around it.

Now with that, So one of the things that I hear about is that people are concerned, vendors are concerned that if I give you my sbam, you're not gonna be able to keep it secure itself. Oh Yeah. Secret sauce.

All right, so, so wanted to hear your opinion on that. That's a great question. The second one is on this new thing, the self attestation, like, you don't really have to give me an ESP bomb, you just need to send me a letter that tells me that you promised that you're doing things that are not vulnerable.

So just wanted your opinions on those too. Um, can I, I always go back to contractual notif or contractual like conversations, right? If I'm gonna be purchasing something from a vendor and I want to get into an agreement from them, um, I like flexing the size muscle.

Like, so when I was at Aetna, I was like, yeah, we got 310,000 reasons and called employees that, uh, you know, and, and a budget that we can say no. Um, not everybody has that luxury, but again, if you're gonna purchase something, you can always ask, right? Um, and, and I think that's, that's number one.

Privacy of information. You know, it's the same CIA triad that you have to deal with, right? You're getting an artifact, it's potentially secret sauce and you have to be able to store it in appropriate place with the permissions and that kind of thing.

Um, there are some tools that do that, um, where you can upload an SBO and it will allow you to share it privately to folks. Um, and then there's companies like Jupiter, um, Jupiter Networks that publishes it online. Well, you used to be there, you told me about this.

But, uh, anyway, great question. Great, Thank you to dj, please. Woo.

show to hear. DJs show every week on Sbam and people using sbo.