Guy Gil, Mend | DevOps Connect 2022

Hey everybody. Thank you for taking the time to join the session today. I'm guy bar Gayle head of products that growth at mend formerly wide source.

And today we're going to talk about how you can reap the benefits of your software dependencies without compromising on agility or on your project security. So what we're going to cover today is we'll talk about dependencies what they are and why you and others would want to include them in your applications. We'll talk about some of the main issues with these dependencies some of the risks that come attached to them.

And of course, we'll discuss how to overcome these issues and how to mitigate the risks because at the end of the day you do want to use dependencies in your projects and then we'll do a quick recap of everything we've we've discussed today. So we'll start with what are the dependency is generally when people hear the word dependency. I find that they tend to think of like third-party packages or open source packages.

This is the most common use of the term. But in my opinion the definition of a dependency is a bit broader than that and it's really anything that you use in your app that you didn't create yourselves. So this of course includes open source packages, but it can also be a Docker image that you've pulled from Docker Hub.

And that's your basing your application off or it can even be code that was written by someone else in your organization on a different team that you're relying on and building on yourself as well. And the average app has well over 100 dependencies. So obviously people see value including them in their applications.

And the reason for that is the dependencies are used to accelerate the business, right? They minimize the time to Market and basically they make your application development faster and you know Would someone take the time to resolve a problem from scratch that's already been solved well by somebody else and the answer to that rhetorical question is that they wouldn't take the time to do that and they will just include that code that someone else wrote in their application to solve the problem. And this is especially true for widely encountered problems such as logging or making an HTTP request.

But you need to make sure that you choose good dependencies and the to manage them when you use dependencies in your application. Otherwise, it could be quite risky. and Yeah, so now now you know, sorry you have some some responsibility now, which is the dependencies.

Yeah. So we'll talk about three main types of risks that are testing Tendencies today. I chose three security related risks and that's vulnerabilities.

It's accrue technical debt and you'll see why that relates to Security in a bit and about malicious actors or malicious dependencies. So we'll start with vulnerabilities as these are the most commonly associated risks with dependencies and what a vulnerable what a vulnerability is is it's basically just a coding mistake that exposes the application to attacks and it's important to know that these are coding mistakes. Okay, so they're not introduced purposefully by someone who's actively trying to penetrate or create a weakness in the application and most of these vulnerabilities remain undetected because nobody's looking through every line of code that was written and inspecting whether it could potentially be used to attack the application or not.

So the vast majority of vulnerabilities remain invisible right there hidden, but Those aren't the Dangerous Ones. They're not the ones that interest us. The dangerous ones are those that are publicly known because attackers know exactly where to go for those and it saves symptomatic to spend time expecting every line of code.

And you know, there are publicly available databases like the nvd which is the the national vulnerability database. It's maintained by the United States government that store these vulnerabilities and make them publicly known to everyone and I think a natural question that arises from that is basically like how does a vulnerability go from Discovery by someone to being publicly known and the answer to that lies in the cooperation between the open source maintainers and the security researchers. So basically a security researcher will discover a vulnerability.

They'll notify the open source project maintainer that a vulnerability was detected in their open source project and generally the open source project maintain a real work to issue a path for that and they'll include Attach in the next version of the project. And once that new version is released then the open source, then the security maintainer will go to the nvd and let them know that they can publish the details of this vulnerability because the fix already exists and it is generally the case that people wait until it fixed exists before they publish the details on the nvd. So, you know knowing this is it enough to just check the dependency you're bringing in has any known vulnerabilities.

It's just what you need to do to be safe and sadly the answer is no that's that's not all you have to do. There's a few other things. But I'll take you through what you should do.

So it's it's four steps pretty much. The first is you have to know what you're using. Right?

This is the most important thing and you need to you need to understand what you're using. Not only in the sense of what the dependencies you brought in directly. But also what dependency is your dependencies are bringing in and this isn't just trivial as it may sound as at first especially when those dependencies are bringing in many more dependencies of Their Own.

And secondly you want to make sure that the dependencies you're using aren't vulnerable or that the vulnerabilities that they have presents an acceptable level of risk to your organization. So once you understood what you're using you've understood if what you're using is vulnerable or acceptable level of risk to your organization. Then you need to have a process for managing the vulnerability remediation and you will find that you have vulnerability.

So this is a you know, a must-have step and for dependencies in the most common method of remediation is just upgrading with the Tendencies version since a path for the vulnerability was already very like released and finally and I guess this is more of an optional step. But once you've gain some trust in your mediation processes, you'll want to automate whatever you can so your vulnerability to remediate it in the shortest possible time frame and most importantly you want to be proactive with upgrading your dependency versions. You don't want to be reactive by waiting for a vul.

Ability alert before you act because then you're you know already vulnerable and I'll demonstrate just how important being proactive is with this side. So about 90% of vulnerabilities are discovered in non latest versions. So a version that's not the latest version of the dependency and the implication of that is that if you're consistently up to date, okay, so you're proactive with your dependency upgrades you're on the latest versions, then you're already avoiding about 90% of newly disclosed vulnerabilities and you can see the data here.

It's from Amanda research paper on ntmcvs in 2021 that about 90% of vulnerabilities are fixed before the cve is made publicly known and only around 9% of them are fixed after the cve or the vulnerability are made publicly known. 7% of vulnerabilities, you're avoiding them. Definitely a best practice.

and there are additional benefits to staying up to date besides just this vulnerability prevention. I think the most obvious is that you get access to the latest features and latest apis of the dependencies that you're using so better functionality for your application. You also get all the bug fixes of your dependencies included so you have less buggy software of your own and you get to avoid what's called zero day fire drills, or at least what I call zero, they fire drills and if you don't know what that is, or you have an experience one then let me paint the picture for you.

So imagine this this scenario. So a new vulnerability was just discovered and it seems to be very important. So think like love for Jake bring for sale these types of vulnerabilities and your customers are pressuring your company for an immediate, right?

This vulnerability makes them vulnerable as well and your company turns to you fix this this vulnerability and you come to fix it and you realize you're 18 months and 10 releases out of Eight and you're faced with an ultimatum, right? You can either upgrade 10 releases without testing and risk of breaking production. Maybe having some downtime for the company.

Or you can you know do everything safely and as as you should but you'll remain vulnerable for however long it takes you to run all your tests and make sure everything is okay before you close the update. So, you know, what would you do in this situation? You can either risk breaking for production or remain vulnerable and I think everyone would agree that would be best just avoid the situation and you can do that by consistently updating your dependencies because then you're not up you're not applying a patch that's you know, ten releases.

You're applying a patch that's one little minor release or security patch and you can do that with a much lower risk than you would of course 10 releases. So Definitely something you would want to avoid. But of course staying up to date can take a lot of work, right?

It's not it's not that easy. Otherwise, everyone will do it. And again, there are five steps that you would need to take for each of your dependencies to make sure that you're up to date so you would have to understand first what dependencies are in your software projects.

So again, this can be open source packages. It can be Docker images or it can even be like internal dependencies. You need to check for each of these different updated version exists.

You need to make sure that that update won't break your build. So you'll run you'll want to run your tests. You want to read the release notes make sure that no like major functionality being changed.

No API is being changed things like that. You'll open a pull request to update the dependency and you know, you have to repeat these steps for free dependency pretty much and this is a very time-consuming process. So on the right here, we have some data from from the strike report called the developer coefficient and it found that the average developer spends around 13 and a half hours per week.

Just Taken care of technical debt and reducing it. So that's about a third of their time. That's quite a bit very time-consuming process.

So you can automate this process. There are tools that can scan your project and identify your dependencies for you. They can check for updated dependency versions.

They can automatically open the pull request for updates and they'll embed all the information you need inside the pull request itself. So that's release notes. It could be the commit history.

It has your test status inside the pull request and something the call like crowdsource merge conference data that you can see an example of here on the right. I'll get into what each of those fields means a little bit later and it also increases your efficiency because you can for example group together like minor or high confidence updates. You can make sure that your team isn't swamped with pull requests by rate limiting them just a few per hour and you can make it.

So the poor requests are opened in times they're convenient for you or your team. For example, maybe only on weekends or after 6 PM or whenever you're not doing your your daily work. And so since I believe that updating proactively is ultimately the most efficient activity you can undertake to improve the security of your repos.

Then I thought I talk a little bit more about to like more advanced functionalities that go on behind the scenes in different dependency updating tools. So the first functionality is crowdsourcing test results. So everyone talks about having good test coverage, but you're not actually going to write and run tests for all of your dependencies because it's simply not feasible.

So if you're not gonna write test for each dependency, then are the test you wrote for your own application robust enough to catch potential problems with new dependency versions. And for most people the answer to that question is it's a resounding. No, so a cool feature that some dependency management tools have is that they aggregate the test results for everyone who underwent that dependency upgrade and that vastly expands the percentage of issues that are going to surface so If I go back to my previous slide, then you'll see that the percentage of tests that pass here in the passing field on the right and those screenshots and you know, this greatly affects the overall confidence score so you can see in the top that 82% of people's test are passing and that makes the confidence score go from very high to to neutral as opposed to the bottom where you can see that this update is passing everybody's test.

It's it's widely adopted by 36% of people using this dependency this version is and the confidence for this upgrade is very high and So the top update like 82% of people would have thought this looks good to merge. It's passing their tests. There's no reason for it not to be good and they would just merge it.

But for people with this advocated data, they know that there is some problematic element to this update and they can take a look and review it a little bit deeper than they otherwise would So moving on to the second functionality It's relatively similar in the sense. It's also crowds First Data but it's crowdsourcing package adoption data on tests. And so let's say that the update is passing everyone's tests, but you still don't feel confident enough to take on this update because the reality is that some bugs can still slip through.

So the second line of defense is waiting for a certain number or percentage of companies using this dependency to undergo the upgrade themselves and you'd want to wait for example a week or a certain period of time you decide on and if people haven't rolled back then you can assume the update is safe. And this is also something that some dependency management tools support as well. So I would definitely recommend that both of these together make updating quite safe and on that note.

I think that like updating dependencies is like going to the dentist. So if you go once every five years, it's really gonna hurt right it's better. You like routine checkups all the time then go once once every long time.

So yeah, remember that and Moving on to malicious dependencies. So these are dependencies where an attacker purposefully inserted code that exposes the application which installs the dependency to an attack. So the difference between a malicious dependency and a vulnerable vulnerable dependency is that a vulnerable dependency is created accidentally whereas a malicious package or dependency is created on purpose.

So malicious dependencies are much less common than vulnerable dependencies, but they're also much much more dangerous. So for malicious dependencies, the general rule of thumb is that as soon as the dependencies installed your application may already be compromised and exposed to an attack. and What can you do about that?

Well here like two recommendations that I have for what you can do to prevent malicious dependencies from entering your system your application. So first you'll want to scan your packages before you install them. And ideally you'll want to do that using the dedicated dependency malware scanner.

There are several like free products that provide this functionality at the point of dependency install and they'll just block the installation if anything malicious is found like an antivirus and second. You'll want to take a small time buffer before you update dependencies to a newly released version. Just to see that people are using the dependency and you know, their world hasn't been set on fire.

and let's quickly with the recap. So today we talked about three main types of dependency dependency risks. We talked about vulnerabilities.

We talked about crude technical debts, and we talked about malicious dependencies or how malicious actors can exploit them. And what can you do about that? Well, three simple things you can scan your dependencies for known vulnerabilities.

You can scan your dependencies from malicious code and you can automate your dependency updates and make sure up to date with your dependencies. Thank you.