The Monsters in Your Software Supply Chain That AppSec Tools Can’t Find with Matt Rose | SecOps Vision 2024

Well, welcome everybody. I'm so happy you decided to join this session. I'm Matt Rose, the field CISO at, uh, reversing Labs.

And the title of this presentation is The Monsters in Your Software Supply Chain that AppSec Tools can't find. Uh, if a lot of you have seen some of the things I do out on social media or seen me talk before, I've been in application security way longer than I'd like to admit, and I really wanna kind of have a conversation today rather than reading slides to you around, you know, some gaps in your application security program associated with software supply chain. But with just some housekeeping to start.

As people have seen my presentation before, I like to kind of insert some interesting pop culture references, movies, songs, what have you, and this is one of my, uh, favorite movies from the past. If you don't know what it is, it is airplane. Uh, we have Leslie Nielsen there, and the famous line from this scene is, don't call me Shirley.

Uh, and Shirley, you don't want to call him Shirley, but, uh, don't call me Matthew. Matthew is something that I have a, a real problem with because that's what my mother called me when I was in pro, uh, in trouble. If the middle name came out, uh, then we knew the bleep was about to get real.

So please call me Matt. Matthew is only reserved for my mother, uh, when I'm in trouble. So thank you for that.

But let's jump into the presentation and hopefully you'll find at least a little bit of it. Uh, uh, interesting. So what is changing in software and more specifically software development?

Um, you know, it's not a presentation if you don't have some metrics associated with a PowerPoint deck. This is, I promise, this is the only slide that has metrics, but I think these metrics kind of map out where it's talking about. Uh, you know, 77, 70 1% of respondents said that DevOps departments are driving software supply chain decision making even more than security teams.

The reason being is software is getting more and more complicated the way it's being developed with CICD pipelines and cloud native development, and then microservices architecture. Things are moving quickly and they're changing. And software developers report, they've experienced purchase delays, uh, for a significant time.

This metric says three months due to code prominence concerns. And what prominence means is, where the heck did this stuff come from? Where did this code come from?

Is it open source? Is it first party? Is it third party?

But let's be more realistic. Something more interesting than just metrics. I like to, again, use graphics.

So let's think about the way software was developed five, 10 years ago where you develop software in a waterfall methodology you'd release every three months. And it was the stone age of development as it compares to today. So we're gonna use the Flintstones.

We have Fred, Wilma, Barney, and Dino all going out to, uh, the movie theater before they put the big slab of ribs on the side of the car and it flips over. But everything was slower. It was made a rock.

We had no flying cars. Speaking of flying cars, software development today has moved into the Jetsons realm. Yes, we don't have flying cars yet in our world.

We're still crossing fingers on that. We have electric cars, so maybe flying cars are the next thing. But we've really supercharged the way we're developing software.

We're releasing it quicker than ever. We're adding features, uh, quicker than ever. And what that's really creating is opportunity for, as I like to call them the hackers or the nefarious dudes, because it's creating very complex ecosystems of software development.

And there's, you know, a little experiment you can have is go into your organization and see if there's one person that understands everything with that application or that piece of software you're developing. So it's always about trying to keep up with not the Joneses, but the way software's developed. So there has to be new approaches for software, supply chain risk, uh, and the identification of software supply chain risk within your organization.

So again, I always like to throw stuff out there and uh, you know, when I think about software supply chain, you think about supply chain, you think about assembly lines. Here's another obscure reference. When I, uh, unfortunately I'm not doing this live for you guys, but I usually like to quiz the audience 'cause I like to be the more obscure, the better in my mind.

So if you can think about a, uh, movie or a a TV show about assembly lines or car manufacturing, there aren't a ton of 'em. I hate to break it to you. There's not a ton of 'em.

But, uh, if you don't know what this movie is, that is Michael Keaton, a much younger Michael Keaton. 'cause this movie is, uh, somewhat dated these days, uh, in a movie called Gung-Ho. And I think it really applies to the way software supply chain risk and software supply chain security is viewed today.

Because the premise of the movie was, it was a US-based car manufacturer that was struggling. Their processes were messed up and they needed to up their game. So they were purchased by a Japanese company that came in with modern methodologies, modern ways to build car, which was painful at first, but ended up working in the end.

Sound familiar? Very, very similar to what we're seeing with the change in the way software is being developed and the way organizations need to not only identify risk, but remediate risk. So to kind of give a picture here, this is an analogy I like to use.

And again, another weird picture for everybody. How many people are, uh, remembering the days when they were, and I know I'm dating myself again when you were sick and this is before the internet and you could watch everything. Uh, I do have drain my beard 'cause I've been, uh, long, uh, around a long time.

But if you were sick, you really watched two things. 'cause there's only three channels at that point. You watched Bob Barker and the price is right.

And then you moved into Days of our Lives. And the, the tagline for the soap opera, if you don't remember it, look it up like sands through the hourglass. These are the days of our lives.

But the analogy here is the hourglass itself. So if you take this hourglass and flip it on its side, I like this as a visual representation of software development, free compilation and software development post compilation. So kind of to map these technologies, map these things together, we're gonna kind of call out some of the left side technologies and the right side technologies of your development SDLC DevOps program.

So on the left side, at first we have the developers working in their native environment, their IDE, their eclipse, their visual studio, their xcode all writing code, the first party code. And then you have your binary, your, uh, code repos, your Git, your GitHub, your visual source, safe, whatever you're storing your code in. You have your binary repositories, and I use jfr, uh, as an example here, uh, where you're storing the binaries associated with the applications and software you're developing.

You also have the tooling itself. CircleCI is a, uh, CI orchestration solution that allows you to put the pieces of your very complicated application or software together. Then you have the build system itself.

This is the MS build logo that I'm using. And then you even have your open source repos, your PI pies, your NPMs. These are all things that you have to be concerned about when you're talking about software supply chain security, because it's not linear.

There's many different ways that software supply chain attacks can occur. And we're getting into some more details in a little bit. But this is just talking about just the sheer complexity of all the things that have to work consistently, securely, and in unison for software to be created.

And then, hey, once you're actually through the process, you're deploying it to your cloud environment, you're deploying it to your container, you're deploying this to your container, uh, to your data center. So thinking about that, look at, you have these two big bubbles on either side. These bubbles have that inflection point, that point in time post compilation, pre-deployment that you're creating the culmination of all your work, your DLL, your jar file, your war file, your iso depending on what development language and platform you're developing in, this is where the final exam for software supply chain risk needs to happen.

The reason being is you could be very, very, uh, effective at looking at your first party code or the code you're writing, your third party code, your open source code. But there's different ways that software supply chain risk can be introduced into the ecosystem of a development process. And we're gonna jump into that.

But this is kind of painting the picture about how software's developed and just the complexity and where a best practice to address not just application security risk, but software supply chain risk. So with that, we're gonna jump into some new analogies. I'd like to talk about the pieces and the whole in this, uh, example and the, the little robots I'm using here, again, I'm going back to my childhood and dating myself.

These are the construc cons and they were transformers. And I'm kind of mapping these to different parts of your application or different states of the application or software you're building. So first of all, very prominent in today's world, our APIs, our application programmatic interfaces, uh, which are basically, let's just say expanding the boundaries of your application and not be a self-contained entity, but being a neural network of many things, working together from, uh, functionality and data sharing, all connected in with secrets and with, uh, APIs and the type of risk these identified are excessive data exposure.

And these are just some examples. This is not holistic by any means. Uh, insufficient logging, monitoring, rogue APIs, authentication risks, so on and so forth.

The next construct con here is, uh, associated with source code in the application security technology. Associated with that is SASS or static application security testing. These are around the lines of WAIS top 10 issues in code, not in a running application, not in a tertiary or or secondary type of, uh, um, element to the application.

Um, but awas top 10 issues like SQL injection, cross-site scripting, buffer overflows. So vulnerabilities is resonant to the code. The next one is a big one.

Everyone's heard about this today. Open source packages and software composition analysis or SCA, the type of risks that you're looking at here are really around open source package risk licensing issues. Pro, uh, providence, that's a tough word for me.

It really is upgrade path to the latest version or auto remediation. But again, this is just working at one lens or one piece of the greater puzzle, and that is the open source code and the inheritance underneath each open source package. Then you have that application in a running state.

We're not looking at components now we're looking at the thing. And when I mean the thing, the running application and you're talking about dynamic application security testing and pen testing. It's, it's very similar the type of things you're finding here that you found in sas.

But sass, again, is looking at source code and DAST is looking at the running application or that manual pen tester with all their tools in their toolbox or all working to find ways to compromise an application with hacking methodologies, with insertion of questions, questionable strings into an input box, so on and so forth. And then you move into that QA arena. This is a kind of a newer technology.

Well, it's not that new these days where you're using interactive application security testing to test the application from the inside out, leveraging functional testing from your QA team. It gives you another lens of, uh, identified risk and, and action items, uh, that are probably a lot more accurate because they're happening from the inside out rather, rather than the outside in. And then the last one, and this is an area just talk about, but there is a production application security solution come runtime application security, self-protection or rasp.

And this is basically a web-based firewall that either sits within the compiled artifact or in the app server itself to act as a firewall, uh, not a firewall, but a web application firewall to basically block potential risks. The problem is these are all looking at that lens of risk, not at the complete architecture. What happens if you put the all the robots together, all the construc cons form one big robot, which is devastator, and devastator is the culmination of all your work.

Just like that previous slide that, uh, kind of showed, um, you know, the left hand and right hand side of the development process. This is what is, uh, kind of at that inflection point of, uh, the hourglass. But this is the best place to really look for the detection of malware.

Did malware get into my application in some way, shape and form? Or did the tendencies for a new novel piece of malware get into my application in some way? 2, what's changed, what's been updated, what's been deleted?

'cause that is a, a clear path to identifying, hmm, something doesn't pass the, uh, the litmus test or the sniff test of what is happening. The creation of SBOs. You know, a lot of times people talk about SBOs or software bill of materials, uh, in many different types of contexts.

But if you're just looking at, uh, uh, an SBO m associated with a software composition analysis solution, then you're only gonna have a SBO m of the open source code. Or if you're just looking at it from a manual process of putting together a bunch of things in a, uh, Excel spreadsheet, there's always that human error factor. So reverse engineering that compiled package, that DLL war, iso, whatever you have in terms of your development platform where your, uh, your code platform, this can give you the com comprehensive bum of everything, your first party code, your COTS code that's been included, your third party code, your open source code, all the things.

And in an industry recognized format, that's one of the key things when you're talking about SBOs, is that you have to formalize on a standard format. And that typically the ones that people focus on today are Cyclone dx, SPDX, and SWID. Uh, that's not the topic of this conversation, but if you have some time, I'd suggest you look at those, uh, main, uh, formats of an SBO M because consistency is key in terms of defining an sbo.

M The other two things to talk about when you're looking at the complete package is, was this package, uh, post compilation and signage tampered with how do you understand if there is risk associated with, uh, tampering of, uh, the package itself? And then the final one, again, applications or software can't exist without secrets these days. They're the key to the lock that unlocks the interesting stuff, the personal identifiable information, the connection to different systems, the trade secret type aspect.

So identifying not only just what secrets are, but which ones are the most dangerous in terms of access to sensitive or protected information. So when you're thinking about your application security program, yes, it's very important. I'm a big proponent of all these application security testing technologies.

They're all great at what they were designed to do, but they were never designed to find the risk associated with open with, uh, not open source, but software supply chain risk, which is really about two things I like to say. It's the how and the what. How did potential malware get into my application that I'm developing, and what is the malware itself?

So when you marry those together, it's like Forests Gump, you know, Jenny and Forrest were like peas and carrots. Again, you have to know the how and the what associated with software supply chain risk. So we'll jump into another friend of mine.

This is, uh, my good friend Wilson. Um, he was Wilson here because if you don't know what to look for, you won't find it. They didn't know where Wilson and Tom, uh, Hanks were.

They were lost on an island. They didn't know where to look, so they couldn't find them. A lot of times supply chain attacks associated with malware are novel.

They're new, they've never been seen before. So how can you look for something that's never been seen? Well, you have to follow the research trail.

You have to follow, uh, the breadcrumbs that potentially lead you down the path to, hey, something's changed here, but I don't know what it's, and the big thing is that malware as a entity, it's not as simplistic as just grepping a code base or a binary repository from malware dll, for example. It's about the DNA of that malware. It may be called something else.

It may be called cat or dog or fish or whatever. I just used a bunch of animals there. But it has that same forensic, uh, capability, the same fingerprints, the same DNA.

And if you see those type of DNA strands, you can go back and say, Hmm, this looks like something is off here. And do a deeper dive with your application security team or your SOC analyst team. So now to get to our monsters, my monster today, he is actually a really nice guy.

This is Sully from Monsters Inc. He's gonna, uh, try and scare you, but we're gonna jump into some examples to talk about what supply chain attacks have existed out there and how they're all different and how they're all, uh, kind of a exercise in understanding risk in different ways. So the first one, three cx.

This is one of the most recent software supply chain attacks. And this was a compromise of the signed package where malware was actually, uh, inserted in the signing process to compromise the signed package from a outward appearance. This looked like a totally acceptable piece of software, and it was signed and, and blessed.

And I always say, when you're talking about signing a package, it's kind of like Game of Thrones where, you know, they throw ravens across, uh, countryside and they're always delivering a message with a, a stamp of the house of Lannister or the House of Stark or whatever. And if it was broken, you knew something was there. That's kind of the same thing with signing a package.

You put the rubber stamp and the, you know, the house of Stark Wolf was on it. So it's, uh, it, it looks like it's effective, it hasn't been tampered with, but there are nefarious ways that you can compose the, uh, compromise that signing process. And that's what we saw here.

The, uh, nefarious dudes, as I like to call 'em, were able to slip something in post signage to compromise the package itself and execute malware. The next one is one of the most famous software supply chain attacks from SolarWinds, the sunburst attack. This is even a different lens where the compromise was at the build environment.

So if you think back to my, uh, construc con slide with, with the source code and the APIs and the running app, if you did the best job in the world to try and secure all those pieces, you would've never have seen this attack because it was slipped in as part of the build. So it was just the, the malware was inserted into the MS build environment during compilation, and it was just packaged up with the application itself. Very difficult to find.

That's why you need that final exam, that inflection point in the sideways, um, hourglass to really, uh, have that final check to make sure everything is copacetic, to compare it to previous releases to see what's changed. Code cov uh, this is the example I'd like to call. Uh, the, uh, if you remember, there was a show on, I think it was FX, called the Americans, uh, where it was, you know, cold War 1980s espionage between the US and Russia.

And there was a lot of cloak and dagger bugging phones or computer terminals to gain access and to listen to conversations. Very similar in this, in that a developer for Code Cov his credentials were compromised and it gave people the ability to insert, uh, bad stuff into the code repo that was then compiled with the application itself. So again, another risk, this is a, uh, user co compromise that allowed somebody just to check into the source code repository.

So then that tainted code, whatever, it's malware or just a executable code, which is packaged up and executed. And then the, uh, the final version here is that tooling aspect where CircleCI was the victim of a software supply chain attack, which basically was after secrets. It was after that neural network of connection from one application to another to compromise secrets.

Maybe that secret is a login to the login credentials or connection string to a database. Maybe it's to, uh, sensitive data from a third party. Maybe it's a functionality to process alone.

This, again, is a different risk. So we're talking about tampering or signing compromise. We're talking about build compromise, we're talking about user compromise, and we're talking about tooling compromise associated with secrets.

These are all different and they're, uh, brand new. It is a new horizon of risk about how these things potentially could happen to your organization. And the biggest thing to think about is this isn't just for software, uh, or application development houses or organizations that develop their own applications.

You can use that final exam for software and applications you consume as well. 'cause taking in doing that final exam, that reverse engineering of a binary or a package can be done for the code you're working on, or the code or applications you're consuming as simplistic as a shipping app or a network administration app or anything. So always do that.

Check that sniff test. And when I say the sniff test, that's my stupid analogy for, you know, when your, your mom, your dad, or your grandmother's, like, Hey, does this milk smell bad? Do give it the sniff test.

I I don't need to do the sniff test. I can do it through reverse binary analysis of the package itself. So software supply chain risk you need to solve.

There's really four areas, and I'm just gonna talk to these kind of, uh, conversationally where, you know, there's always a risk and compliance, the executive order out of the White House, the follow on memorandum, talking about mandates of cell, uh, self attestation and SBOs and NIST coming out with different standards associated with software supply chain. So you always have to be in line with that risk and compliance software, Devon release process from the open source repost to the tooling itself. When we use CircleCI as an example, the release, uh, of, uh, the package itself and the dependency analysis, uh, because it's, you know, very typical that open source packages or even first party coder inheriting from other code to understand what the dependencies are for that to work.

And then automation of software, supply chain security as part of that DevOps CICD pipeline. Because if you're trying to do this manually, uh, I'm just gonna be totally latent with this, you're gonna fail because you can't keep up with the speed of software development that these days, the IT procurement and software. I kind of foreshadowed this in, in on the previous slide, but, you know, third party caught software selection, you know, is the software that I'm purchasing or using in my organization secure?

Um, one of the biggest things I always like to say here is even if you go through due diligence and security questionnaires, not many people are gonna say, Hey, my software is, uh, riddled with holes. Don't buy it. They're always gonna say we're great.

It's the responsibility of the consuming organization to do their own due diligence and always looking for reoccurring automatic software updates. If you know, you're, even if it's, you've already purchased the software, but it's, you know, changing a version or they're like, oh, you got a bug upgrade to this version, that's exactly what happened to SolarWinds. It was basically just a regular upgrade in the version and that's where the risk was entered.

And then all this information that you're collecting, you can do deeper dive with your SOC analysts. So doing that malware, uh, manual analysis, the self detonation, the triage and response or the threat hunting with the information that you've received from that final exam of either the created or consumed package. So food for thought, when you're talking about software supply chain security, you have to really think about four main areas.

I really feel that the only way to have a comprehensive view of risk and do it correctly is to look at the packages through proprietary recursive binary analysis, crack that thing open, normalize it, and then do your analysis to the lowest levels because that is what's being deployed. That's what you're selling or that's what you're deploying to your customers. Always when you're looking at application security testing technologies or uh, software supply chain, always think about what is the intelligence behind the product.

Most people say, oh, we support all these languages. We support all these metrics or all these standards. How long has that repo been growing?

How much has it seen out there? Because it doesn't matter if you can recursively, uh, reverse engineer a binary, if you're not using any intelligence behind it, you're not gonna find a lot of things. So always check that, uh, ability to execute in terms of the research or the intelligence behind the product.

And then really being able to execute at speed and scale is the best way to put it. Uh, I've been in sast, uh, for a long time or static application security testing. A lot of people know me from my time at fortify or check marks for many, many years.

One of the biggest hiccups was you couldn't scan a specific language or it just took too dang long to do it. So if you can actually not tip the apple cart, not be a barrier for success by being able to scan many different large types of files in many different languages, you will be successful. And then support multiplies enterprise teams, you know, with standardization with NIST and SBO M and secrets and malware.

These are all the things that are basically, uh, lined up to that previous slide from risk and compliance, DevOps, AppSec it, and then your SOC analyst. So I know everybody probably didn't want homework, but here's a little homework assignment for yourself if you wanna do it. There's, you know, you won't get dinged if you don't do the homework assignment, but software supply chain security is a very hot topic.

A lot of people are talking about it. A lot of organizations are saying, Hey, we address software supply chain and those are from the tooling vendors or from the open source scanning vendors, the software composition analysis vendors. And they do a good job.

But when my opinion is that it's the who and the what, again, how does potential malware get into my application? What avenue does it take to compromise my build or the build of the software I'm consuming? And what is it, what is the malware?

The homework assignment is to go to all these AST vendors, whether they're SAS or das or SCA or I asked or Rasp and kind of do a little research, see if they mention malware. 'cause really all those attacks that I mentioned previously are all associated with some sort of malware getting into the deployable package or the upgrade of a version. So do your homework and, uh, we'll, we'll thank, uh, Mr.

Stark here, uh, for his, you know, winter is coming or homework is coming. So with that, that was the presentation. Thank you very much.

Again, I'm Matt Rose, uh, field siso at Reversing Labs. Uh, check out, uh, on LinkedIn. You can look at my pro, uh, my profile, check out my Reversing Glass episodes, which are a video glass board session that goes into a lot of detail about, you know, topics of today.

Thank you everybody. Hope you enjoy the rest of the conference.