Sarit Tager – DevSecOps: Opening the Fast Lane for Secure Deployments – DevOps Experience 2024

Hi everyone. I'm sorry, Dager. I am the view product management for cloud application security in Palo Alto Networks, part of the Prisma Cloud, uh, product.

Um, I wanna discuss today a bit about how we open a real fast line between security and development. Uh, a bit about my background. I was a developer and engineer, and I come of, kind of come from the, uh, pain side of things when I know how developer actually get blocked by different security issues.

And then they sometimes need to come back and fix things that they may have already deployed like months ago. So I really wanted to interest, like my, uh, thought on how we can really, uh, make it much more easier for developers to work with application security and make sure that all these DevSecOps is really, uh, streamlined as we want. Let's talk a bit about applications.

So we know application is the core of our business. We know that it's kind of, uh, became much more complicated in the last, uh, few years, especially with the cloud. It's not just about your code.

You have an open source code, you have different production environments, you have different users, you have different APIs, um, and we have different storage, and we have different compute capabilities, which means that an application became something that is hard to, uh, actually, um, maintain, secure and make sure that we don't have any breaches on, on production. If we look at the current cloud attacks, um, at least 74% of the cloud breaches are caused by insecure code. Think about it.

Well, it makes sense 'cause application are code, but almost, um, 74% of these applications are really being breached and are sort of something that should have been blocked in the code and not reach protection. And then about the average cost of our, our cost to, to the companies that need to fix it. It's about four and a half, uh, million dollars.

And the amazing part that I think the most, uh, scary part is the fact that it takes about 120 days to actually fix and redeploy, uh, the issue that was found. Think about the fact that if an issue was funnel production, the second, the, the next time it'll actually be able to be fixed is about, um, four months later, which is a huge amount of time for agile development and for developers to walk and understand what they actually deploy to production. If we look at the, where the risk relays, well, we don't, we kind of have it everywhere.

We have it within the code. It can be on our open source, it can be on our, uh, proprietary source. It can be on, um, our terraforms.

For example, if you do an IC configuration, you may find secrets. And even it can also happen on your ecosystem, like your repo or pipeline may not be configured correctly. And this will create, uh, problems within your environment.

And then even if we go into the production, then everything can be a, a potential source for security issue. It can be on the misconfiguration of the cloud infrastructure. It can be on your data, it could be on your identity, on your API on everything that basically exists on production.

So security issues and risk can come from so many different parts of your cloud and applications, which makes it much more difficult to actually make sure that, uh, the production environment will be safe and secure. Just looking for the code, uh, we talked about it a bit, uh, earlier. We have the problem within the code, which is actually in the code we write.

We have the code that we're being bring in. You know, the developers use a lot of open source components to, uh, uh, speed up the development, make sure that you don't, you don't, uh, write things that are already implemented by others. And, you know, there is a power for open source to be used, but it also brings a lot of problems into the either the system.

And then we also have a problem on the way we configure the pipelines, the repos, you know, there are cases in which, uh, there are, there are some secrets there that are being left on our pipelines on our CICD models. And this can get into production and then get access to a very sensitive data. So even if you just look at the code, we can see a lot of places in which we have, uh, potential, uh, risks when the, how did this, uh, this actually happens, like how code issue really get to production.

So we know that for cloud, especially in the speed and the agility that is currently being deployed for all application developers build, want to build very fast. I would say that even if the use of AI and, uh, code generation, the even more code is being built into the environment, and usually developers are being measured by how much they actually put in production, like their product and their velocity. And not necessarily about, uh, security, which means that security will get, uh, probably a lower priority in fixing things.

But let's say for example, for bucks. Then the second one will be, um, security teams select the visibility into the entire development pipeline. Like, I can be a developer, I can run code, I can do a me request, and then the next time anyone from security will actually see it.

It'll be on my build or before production, or even in some cases, only when I build the image that needs to go into production. So they don't necessarily have all the information to understand where, where scanners are being set, what type of guard wires we have on each of the scanners. So this is very complicated for them to understand it.

And the last one, um, today, uh, most of the current guard rails are just to one, like remember yourself thinking about, okay, I cannot bring a critical CVE into my environment. Great, but my open source is using, uh, but the open source I'm using is actually something that has a critical cv. So should I block it at all?

And I think one of the problem is that today it's either, uh, you don't put anything on production or that, um, uh, or that you bring everything to production, but carries a lot of, uh, security risks. So we really want to make sure and try to figure out how to do it more granular, uh, guards, uh, to our environment. Think about, let's try to figure out if this trip is really going to production before we actually stopped things.

Um, talking about visibility and I, I mentioned before, the amount of, uh, different frameworks, languages, pipelines, and technologies that are actually deployed on your environment is almost, um, something that you cannot really maintain. You have to understand exactly like there are some developers that want to write in, uh, Java. Some will be in Python, some will be in OJS, and they will use dockers.

Some will use VMs. There are so many different version control system. There are so many pipelines.

All of these different systems brings risk into the system. And trying to figure out what you actually see within your environment is super complicated and also does not allow the security people to actually understand what they need to block. Think about look for J, which I think, uh, look for Shell, the actual CV that was, uh, found and kind of create a lot of noise.

Uh, two years ago. Um, the most, uh, troubling question was, am I affected? Like, is my production um, not secure because of, uh, this, uh, logo for shell, uh, vulnerability?

And it should be simple answer yes or no. But basically you have to do a lot of dig, dig deep in, uh, dig, dive into, uh, understanding whether I'm vulnerable. Like where are these workloads are being deployed?

Am I using the vulnerable, uh, version? Um, is it really reachable on my environment? Can I understand which workload I need to prioritize first?

Can I do some guardrails that will be kind of a perimeter, uh, blocking? So all of these questions are very difficult to answer in today's world. Like you don't really necessarily know exactly if you are, uh, vulnerable to this, uh, um, to this new, um, uh, this, this new risk that was found, uh, on the wild.

And the reason for this is, again, the lack of visibility within your tools and the different scanners. If we'll talk about what, uh, we believe should be the case, um, I will talk about prevention and how prevention actually saves money. So first it, I think it's, it lets, um, it's, uh, kind of not, uh, annoying the developers.

Like if you get an understanding what you need to fix while you write it, the code, it'll be much more easier than trying to go back and, uh, and, and fix the code that you wrote months ago. Um, it's also also been impacting sense of cost and timing. We talked about the 120 days of, uh, uh, trying to bring something into production after you have to fix it.

But think about all the, uh, cost it happens for developers, uh, to fix issues that far in production. And you have to get back to a versions that you probably don't have in your environment today. So this all cost money and takes too much time.

I will give, uh, just an example of what happened when an issue was funnel production. So issue was funnel production. Okay, we can implement a patch, but we need to fix it, right?

It may be that the, uh, patch is good for now, but it's not something that, uh, can stand for a long time. The second question will be, okay, I need to, to like to locate a source of the risk. Like where is the risk coming from?

Is this came from my code from an open source, from the pipeline, so I understood it, but then I understand, I need to understand who is the owner? Who is the person I need to call to be, to be able to fix this, this issue. It's not easy as we think.

There is no automatic way of saying, okay, you are the owner. Usually you get into, uh, the, uh, VP engineering or the security champion. You will not get into the developer.

The next one will be developer has to do a review to code. He has to understand whether this vulnerability can be fixed, whether you need to change the version of an open source or change its code. So it actually brings him back to a code.

It may, may not, uh, have seen for, uh, several months. And then he has to do a merge request to the system, and then someone need to review it and then push to production. And this end loop, I'm, I'm sure you got tired for the, from this as well, is an end loop of trying to catch up things in production and try to fix them.

And basically we see the developers, uh, struggle to, uh, have so many problems from, from production that they have to fix after the fact. And they know that an issue found in production, it's much more urgent than something that was found in the developer environment before it gets into the main branch or something like that. So we must find a better way to start risk from the beginning.

And this is why we believe prevention is the best way to do it. Um, just for a, uh, um, why could, today prevention is a bit challenging. So think about the developer that writes a code, do a ME request, and then the code violates the policy and then the bill is being blocked.

And this is my last, uh, day of the sprint, and I have to, uh, submit my, uh, um, my code into the, to, into the version and make sure it goes into production. But my bill is blocked. And when I look at it at the security issue, I don't really think that it's an an issue because all I'm doing is actually writing something for testing.

So why am I being blocked by a security policy that is relevant only for production? And even these small things of not being able to say, this is for, this is for testing, it's not for production. This is not reachable for my code, it's not relevant for MyCode are super complicated today, in today, um, in today's solutions.

And you have to wait for some, um, uh, I would say, uh, manual approval for someone that would say, okay, I'm okay with that. You can wave this, uh, um, this block build, which I, in general, this is kind of the, the, uh, the comparison between trying to prevent from the beginning and prevent it too much. What we offer is actually, uh, to save to, to kind of solve all the problem we talked about is first have central visibility.

Like an upset person must have one place that you can see all the problems. By the way, developer as well, developer want to see all his problem, whether it's, uh, open source secrets, IIC problems, or, uh, or static code analysis problems within the same place. The EC will see it within a platform.

The developer will see it within an id. The second thing will be, uh, let's implement a real accurate gut risk. Accurate gut risk means that I will check only for things that are relevant to my code.

Like if the code is not reachable, then then don't block me. I cannot do anything. Like, I don't need to fix this, uh, specific vulnerability if the code cannot be reachable.

Um, don't get me into something that doesn't have a fix. Like it's, it's been blocked. So how exactly will I go and fix it because I don't have a fix?

Maybe I need to move the entire, uh, open source package. And then the last thing will be about a better communication between, uh, AppSec and developers. Instead of we just the apps setting the policy and the developer complaining.

Maybe kind of create something that both developers and AppSec can contribute to and understand that working together will bring them to have a much more safer production, but also, um, allow the, the developers to focus on new, uh, features, uh, development and on new, uh, innovation to be added instead of going back into, uh, their, uh, code from like two months ago. And then fixing things that were found in production. Now, one of the things which is a key factor for this is bringing contexts.

I talked about things that we can bring from the code, like reachability, debt code, uh, uh, sensitive code, the code that being used multiple times or uh, um, or things that are more relevant to testing environment or production environment. But the second thing that we need to bring is, um, a code context, a cloud context, sorry, the cloud context is about bringing information from the production environment. For example, is my airport really going into production?

Is it going into something that is being deployed like a workload that was deployed? Is this workload even exposed to, uh, an external source? So the, all of these questions are super important to be able to, uh, give the developers a better guard rails and not just block them on everything.

We have to find a balance between having a safer production environment and making sure that we have, uh, a good, uh, velocity within the developers. So instead of doing proactive, uh, instead of doing reactive security by finding all the things that are in, in, in production and then asking the developers to fix, we want to do a more proactive security, meet the developers and the security within their environment, within the IDE, within the pull request, and give them all the information to be able to understand whether an issue is really a problem that can, can create a risk. And the second thing will be provide remediation options.

Give them the suggestion how to really remediate them and think about the fact that they don't have much time, but they do want to fix things and make sure that they are being, uh, um, handled in the most, uh, in the most effective way instead of just trying to, to solve them after the fact, after they go into production. So key factor is, is the context from both runtime into the code, from the code itself, from the different ecosystem environment. For example, configuration of the repoint configuration of the CICD pipeline.

Everything that will make, uh, the developer and the AppSec life much more easier. The AppSec will be calm, knowing that there is no, there are no risks coming into the production. And the developer will know that is not being blocked, just, uh, uh, for the sake of having, uh, policy security policies.

But you will understand that if the build was broke, it's for a reason that it has to be fixed. The issue really need to be fixed. So context will be the key for, uh, effective AppSec.

So I think, uh, just to summarize, uh, we have to have a fast lane between security and developers. They have to be friends and work together and make sure that we have both of them happy, uh, with the, with the status of the security on the production, and also with the fact that they don't wanna lose any, uh, days trying to fix things that they're already, uh, put in production months ago. So thank you very much.