Making Your Software Stack Security-Aware | DevOps Connect: DevSecOps 2023

Hi everyone. My name's Jeff Williams. I'm the CTO and founder at a company called Contrast Security.

We provide an application security platform, and today I'm gonna be talking about making your software stack security aware. It's a really powerful technique, uh, and I think you're gonna like the effects that it has on your application and API security programs. Um, just a little bit about me.

I've been in AppSec for over 20 years. I helped to found OS by wrote Theos top 10, and I've done, uh, zillions of pen test and code reviews over the years. I've helped a lot of companies with their AppSec programs, and so I'm really excited to share this, uh, relatively new approach with you.

So, uh, I do think that app and API security has changed a lot. I think this is the most exciting time in application security in history. Uh, in the early days, you know, we did a lot of manual stuff.

Then there was a long era of what I'll call big process. It was like you have to do a lot of things, a lot of activities, but we really didn't end up securing very much stuff. And in fact, you know, the applications during didn't get a lot more secure.

They still have the same kinds of top 10 flaws that they had, uh, you know, even almost 20 years ago. And then recently though, uh, I think things have changed a lot. There's a lot of, uh, technological changes in the market, faster apps, faster developments, uh, new kinds of applications, and government has been, uh, pushing into these markets.

Governments around the world actually have been demanding more transparency, uh, more rigor in application security, and they're even demanding that, uh, organizations start attesting to what they've done to secure applications. So I, I highly recommend getting your AppSec program healthy so that you're ready to be transparent about it when, uh, when it's demanded. So this is a little bit what our industry looks like right now.

As you can see, there's a lot of different fragmented pieces of the AppSec market, and I doubt that many of you know what all of these things are. Uh, I didn't actually. And the good news is there's really only a couple of things going on here.

Some of these products are designed to find vulnerabilities, things like, and the newer one I asked are designed to help you find vulnerabilities, I guess mask for mobile applications as well. Also S c a, that software composition analysis to find issues with libraries in your applications and APIs. Some of these things, uh, some others of these things are about identifying attacks and preventing exploits.

Things like WAF and rasp and web application and API protection. Uh, things like cloud workload protection. Those things are about attacks in operation.

The rest of these things like ASOC and A S P M are, are kind of meta, they're tools to help you overcome the weaknesses with these other tools and correlate them and, uh, and, uh, orchestrate the tools. It was very complicated. I don't buy into a lot of this.

I want you to focus on two things, find vulnerabilities and get 'em fixed fast, and make sure you have visibility into who's attacking you and the ability to stop those attacks. That's it, that's what, uh, stack hardening is really all about. When we call it security aware stacks, that's what we're talking about.

Um, I'd say all those tools put together really have a, a number of issues. Many of the tools generate lots of false positives. The scans take a really long time, like it could be hours or days for DASK scans to, uh, and, you know, hours for DA SAS scans to run.

That's way too long for modern development. With a pipeline that only lasts 15 minutes. Uh, you have to kind of finish within that window.

And ultimately, I think they're delivering pretty weak assurance that your applications are thoroughly tested so we can do better. So some of you may know about A S L R, and that's address based layout, randomization and data execution prevention. This is really the first runtime protection.

And what these technologies did is harden the stack so that buffer overflow, uh, attempts would be either impossible or much more difficult. And it's kind of built right into the, the operating system and the, and the way that you compile code so that when your application runs, uh, it's protected against being, uh, exploited with a buffer overflow. It's powerful.

It's built into the stack. And the cool thing is by hardening the stack, even if the developer makes an inadvertent mistake, like they use an unsafe method in CRC plus plus, you're still not exposed. Uh, so the, it's a very cool model and it radically changed the landscape for kernel exploits in the early two thousands.

So the big question is, can we do this? Can we replicate that success for web apps and web APIs? And I think we can.

Here's how we do it. So imagine this is your stack. You've got a runtime platform like the JVM with C L r.

On top of that, you're probably running a, uh, an application or API server. On top of that, you're running, uh, application framework or an API framework, and you may use a bunch of different open source libraries. And then on the top of the cake is your custom code, the business logic that you've implemented.

So what we can do, and, and this is really interesting, so the, this, this stack has a bunch of dangerous functions in it. They're really powerful. I mean, they're useful functions, but they're also kind of dangerous, right?

Things like, let me start a native operating system process, like a runtime dot exec, or maybe let me make a database query. Let me open a file. Uh, let me parse an XML document or evaluate, uh, uh, expression, language expression.

All those things have potential to be really dangerous. And these dangerous methods are not protected at all. They're just like any other method, developers can call them with no special magic, uh, attackers can target them because there's really nothing that protects these methods.

So what we can do is, as these methods load as all these, this is all really just libraries. As all these libraries load into memory, we can instrument them. We can harden them with trust boundaries that do a couple of things.

They can first, they can ensure that developers use these functions safely. So if, uh, a developer uses an application in an unsafe manner, we can warn them, Hey, you're about to, uh, make this sequel call with untrusted data, or you're about to parse an XML document without turning off doc type processing, things like that. We'll look at all these examples in depth in just a second.

And in production, these same trust boundaries can identify attackers who are targeting these functions. And so even if the developer has used them incorrectly and that application makes it into production, we can protect these, these applications by preventing attackers from exploiting vulnerabilities that made it into production. And we're talking about both vulnerabilities in your custom code as well as vulnerabilities in libraries.

Things like, you know, a custom SQL injection and a vulnerability like log for Shell or spring for Shell, or something that's in one of the libraries or frameworks that you're using. These techniques apply to both of them at the same time. So let's take a look at how this works for SQL injection.

Oh, and by the way, this stack hardening, the Gartner analysts would call this, I asked and rasp, this is interactive application security testing and runtime applications, self-protection. But that doesn't really matter because those are artificial categories designed to, I don't know, cause, cause uh, uh, the proliferation of vendors in the apsec market, it's really the same underlying problems. So that's what we're gonna focus on.

So here's for SQL injection. Let's take a look at how this works. So, uh, if you're a developer and you're building an application, maybe you create a search function or something that's gonna access a database, imagine that you take, uh, some untrusted data, in this case, this parameter, Jeff, that's gonna go through your custom code somewhere.

It's gonna build that into a query. It's gonna make a call to the SQL library, and then that SQL library is gonna hand that, that call off to the database where it's gonna execute and return your some data. Now, if you don't escape and or parameterize that data somehow, then you're gonna be vulnerable to SQL injection.

So how can we make that more automated? Well, the first thing we could do is when the developer's doing normal testing and they send in the word Jeff, that data can, when we see that flow into the SQL Library, we could warn the developer. We could say, Hey, you just invoked a SQL query with untrusted data that wasn't properly protected.

We can give them instant feedback on the code that they're building and testing. That's really exciting. That means anybody can test their own code.

They don't need to know how to exploit SQL injection. They don't need to know how to run scanners or configure a DA or SaaS tool. All they need to do is their normal test process, and they can get an instant warning on the, on the fact that they use this SQL call in inappropriately.

So that's really nice. That's a fast way of getting developers to write more secure code. But we also know that there's gonna be applications in production that have SQL injection vulnerabilities.

Maybe they were built a long time ago, maybe, uh, you know, developer didn't use this process and checked in code, whatever. So we wanna know that we're protected even if an attacker attacks this application. So imagine an attacker sends in an exploit for SQL injection like single tick or one equals one.

That data would flow through this application, end up in that SQL query. And you can see how it ends up in the query here. It actually modifies the meaning of the query so that the attacker doesn't need to provide a valid password.

They can log in as anybody. It's pretty important security to have here. What we can do in this application is, uh, we can h put this trust boundary in place so that we can detect that the attacker's data modified the meaning of this query.

That's really powerful because th that should never happen. Attackers should never be able to change the meaning of your queries. If it happens, then we know it's an attack and we can block it.

We can throw an exception inside the sequel library to jump over the query and we can notify, you know, we can create logs or we could notify this to, uh, a SIM and let them know, Hey, this system's under attack and this attacker's, you know, using an account that's compromised or they're trying to hack into the application. So that's the essence of this with one simple trust boundary here, we can both identify vulnerabilities to developers very accurately and very fast, and we can alert when systems are under attack. This is the, this is to me is the path towards eliminating SQL injection.

So let's look at how this works under the covers. So first here, uh, I want to tell you how this instrumentation actually happens. This is using the same technique that tools like AppDynamics and New Relic use for performance management here, we're just using them that same technique, but for security.

So this is the MySQL SQL driver. It's a little snippet of the code in there. And let's imagine that as this code loads, we just dynamically instrument this method with a trust boundary.

We're gonna insert a static method called here that does some checks from within this method. This is very, I mean, this is widely used technique. You're almost certainly using instrumentation under the hood in your applications already.

And here, uh, we can insert this trust boundary seamlessly and very fast as the application starts. So you don't have to change anything about the way that you build, test, or deploy your code. You don't have to change your code, you don't have to do anything like that.

You can just add this, uh, engine to your workloads and all this will happen automatically. Now, here's what that check actually looks like. Uh, in this case, it's really high performance, very lightweight security check.

Actually it does three things. The first thing it does is it analyzes this MyQ L library to make sure you're using an up-to-date version that does all the kinds of S C A activities that, uh, you know, you buy tools to do. Um, so we can do that automatically here as well.

The next thing is, it's gonna check to see if the data in this, this sequel query that was coming in is untrusted. If it is, we can report as a vulnerability because untrusted data shouldn't flow into this method. And the last thing is, we can check to see if this query is actually an attack.

If it's an attack, then we're gonna report the injection attempt and we're gonna throw an exception to skip over. That'll skip over the rest of this logic. And, uh, you know, not execute the query, not send it to the database.

So that'll keep us safe. So this is really powerful. We can really solve SQL injection by putting the, the checks in exactly the right place.

So thi let's think about that. One approach to SQL injection would be to, you know, look at a, your custom app and API code. You've got, you know, lots of user inputs coming from all over the place, flowing through your business logic among a zillion different data flow paths eventually ending up in tons of queries.

I've seen apps with thousands of queries and we, you know, to fix SQL injection, you have to go through and make sure every single one of these is correct. And it's difficult. That's why it's been 25 years we've been trying to solve SQL injection.

We haven't made any progress. But look what happens if we put this fix in the right place, we'll put a trust boundary right into the MySQL library in one place. Well, actually it's like four or five different methods that actually send queries to the database, but it's not a lot.

We can put these trust boundaries in place and solve all of those SQL injections all at once. This is the really, this is powerful. This is the way forward to scaling application security by getting the fixes to these vulnerabilities in the right place.

So I'll just show this on a couple of other kinds of vulnerabilities. This is unsafe decentralization. This was one of the causes of Log for Shell.

And it's, uh, it's dangerous. If you're decentralizing untrusted data, there's not a lot that you can do to, uh, to fix it. You have to really have some protections in place.

So if it's, if you know, during development, if we see untrusted data go into the decentralization engine, we can warn immediately that you shouldn't do that. You should figure out a safer way to, uh, get data from users. And if you see an attack come in, uh, this data flows through and actually tries to trick the ization engine into calling things like runtime dot exec, you can stop that.

So in this case, what we're gonna do is we're gonna block dangerous calls from happening during ization. Uh, the attack technique is called gadget chain and we can prevent it from doing harmful things. Another example, expression language injection.

This is the root of spring for Shell and a number of and struts and a number of other vulnerabilities over the years. This is a little interesting. Uh, you know, same kind of pattern here.

If there's untrusted data being evaluated in the expression engine, that's dangerous cuz the attacker could send in a malicious expression like this and do whatever they want. We can stop it by, uh, same kind of technique as we used last time. We can block dangerous calls from happening during expression language evaluation.

Uh, and that works even if this data comes from a trusted source. So in this case, the attacker was able to put some data into one of our backend systems. It flowed through mq and it ended up going to this api.

And then the API evaluated this expression. So from the the app's point of view, this data was trusted. I got it from an internal system, but it still had an attack in it.

So we can stop that as well. A WAF or, you know, some kind of perimeter protection could never do this. But here we know exactly what's going on here in context and we can stop this attack.

Let's take a look at another one. This is X X Z, uh, XML external entities. And, uh, this happens, it's kind of like unsafety sterilization.

You see, here's a trusted XML document comes in. Uh, if the XML parser doesn't have dock type processing turned off, we can warn the developer and say, Hey, you need to turn on, uh, turn off dock type processing, or else the attacker could exploit the parser. Here's an example of an attack that's like that this, the attacker says, uh, I'll send a malicious doc type that defines a new external entity and then resolves that entity in this case, uh, including the Etsy password file in the resulting xml and then probably sending it back to the attacker.

So we can stop that. We can block this. If, uh, this attacker sends this document in here and we see that, uh, dock type processing is enabled, we can prevent that from happening and prevent the exploit.

This is much more accurate than trying to block these attacks at the perimeter. You know, at the perimeter, all you would see is a bunch of xml, uh, you know, special characters and quotes and end brackets and things. It's very difficult to identify these attacks by looking at network traffic.

Here we're seeing exactly what the parser sees. So we don't make mistakes. We don't over block by blocking things we shouldn't and we don't under block by missing things that we should have blocked.

So this technique doesn't require a lot of configuration like WAFs and it's really very efficient, uh, in terms of C P O and memory and things like that. Uh, last example is we can solve RRCs that use class loading. This is a very common technique for attackers, uh, to use the class loading process to get, uh, uh, native commands to run.

And the cool thing is, normal developers don't use reflection to access class loaders. So we can stop the use of reflection from accessing class loaders. Developers don't need that because they can always get ahold of class loaders directly.

They don't need to use reflection to get in, but attackers can't do that. So they use these tricks to get ahold of the class loader access, uh, you know, force other classes to load and then get them to run Here we can just stop that. So this eliminates whole classes of rcs that have, you know, really most of the major exploits over the past 10 years or so have used a variant of this technique.

And we can just put that to bed. So it's very much like, uh, A S L R and depth in that we can stop huge range of attacks, uh, with a simple technique that really has almost no performance hit here. So those are just some examples of how you can use this technique.

I want to back up here a second and go a little higher level and talk about how this changes your AppSec program. I want you to imagine that you just add this kind of instrumentation to your software stacks. You can do this at scale, you can push it out with the Kubernetes operator.

You can make it part of your gold app server. You can make it part of your standard containers that you use. It's easy to push out across your, and then immediately as you do your normal development and testing dev on the development side, developers will get instant feedback about what's going on in their applications and where they have vulnerabilities.

They can fix those problems, retest and get instant feedback that they fix them. So this is a real-time AppSec feedback loop that doesn't have to go through an AppSec team. It it's really, you know, you can communicate right with development teams and get them to find and fix their own problems, which is much more efficient and much more scalable than sort of traditional AppSec programs.

On the operations side, same kind of thing here. Uh, we can automatically detect, uh, attacks and probes and that telemetry can get shared with operations teams immediately so that they can investigate vulner, uh, uh, attacks, see exactly what happened and take appropriate action. That intelligence that they get, you know, who's attacking us, what attacks are they attack techniques are they using, which systems are they targeting, that information can also flow right back to development teams to give them that threat intelligence to allow them to do threat modeling and focus on the things that really matter.

And all that has to happen here is we just have to add this security awareness to our software stacks. The rest of this happens automatically as part of the normal development and operations process. The security team, because they now don't have to be part of the critical path of finding and fixing vulnerabilities and managing scans and configuring security tools and things.

Now the security can team can focus on more higher level issues. They can define security policy, set security architecture. They can ensure posture, they can help out teams that need support, but they're not part of the critical path to production, which makes this whole process run much more smoothly.

We've seen some tremendous benefits with this programs. So with this, this stack aware kind of program. 2 per month.

With this stack aware, uh, process we're seeing that reduced down to less than one new vulnerability per month, which is much more manageable. And the remediation time for those vulnerabilities has dropped from 290 days with static analysis tools. Uh, it's 315 days by the way, with dynamic scanners down to just three days.

This is across hundreds of thousands of applications and many very large enterprises. So we think we've really cracked the code, so to speak, on making application security work much more efficiently. Now, if your program is introducing vulnerabilities at a high rate and taking 290 days to fix them, what we see in almost every organization is that the rate of finding vulnerabilities keeps going up.

So we we're finding more and more and the rate of fixing them doesn't keep pace. So I know many organizations that have had huge vulnerability backlogs, hundreds of thousands of vulnerabilities that they're not triaging and not fixing because they're, you know, there's just, the process is broken here in the companies we work with. We're seeing that organizations are keeping up with their backlog.

They've, uh, found a lot of vulnerabilities, they've fixed them in a relatively short amount of time, and so they can kind of stay healthy by keeping close to where they're supposed to be. This represents a very healthy application security program. And you can see the line is, is tailing off towards the top here.

That represents learning cuz now developers aren't introducing vulnerabilities at such a fast rate, and that's really the most cost effective way to do AppSec. You can also see that, uh, many organizations, uh, really have no idea what's going on in production with respect to attacks and, uh, customers using the stack hardening approach, they see about 13,000 attacks on their applications per month on each app. 1% of those are what we call probes.

They're attacks that never reach their corresponding vulnerability. Like a sequel injection that never reaches a sequel query or an X X E attack that never reaches an XML parser. 9%, the ones that actually make it to a real vulnerability.

Those are the ones where we, we need to do something instantly and, uh, remediate that flaw. So, uh, as I mentioned, you can do this at scale. Uh, you can push out this instrumentation across your various platforms, your various stacks in all different environments, and quickly achieve this real-time visibility across both development and production.

So because this is a distributed approach to security, it's very zero trust, like actually, and it means that you can scale, uh, to pretty massive scale quickly. Let me quickly show you how this works in the real world. So, uh, I have 45 applications that I've instrumented and made.

I've hardened their stacks so that they're now reporting to me this telemetry that's actually across 76 different servers in different environments in q dev, QA and production. So I'm building this picture. I can see all my libraries and which ones are vulnerable.

I can see all my vulnerabilities in my custom code and I can see that I'm under attack here. And I'll just demonstrate very quickly how this works. So I've got this application called Spring Pet Clinic here.

This, uh, is a pretty typical spring boot application and, uh, I can just demo this for R s a DevOps, uh, I'm, I'm testing SQL injection here by using normal data. I don't have to attack this, I don't need to know anything about security. I can do this manually or in an automated way and I can just use this application normally in the background.

This hardened stack is now doing the security testing for me in the background. And, uh, I don't have to do anything differently. So this is really convenient.

I won't test the whole application here, but you get the idea. Now, when I go, I'm gonna drill down into that application. I'm gonna, you know, go through, uh, this application inventory.

net core node, Ruby, Python, go Kotlin scholar and php. Let me drill down to this, uh, pet clinic act that I was just using. And you can see I've got a nice picture of the, you know, what's going on in each of my environments.

All live, always up to date. And I'll drill into the vulnerabilities here and I can see just in the past four seconds, we discovered a couple of vulnerabilities here. Uh, you know, actually three or four of them.

And I'll take a look into this hibernate injection. This is a really detailed finding from the hardened stack. This is more than what you'll get from static or dynamic application testing tools.

Here you can see the data I typed in R S A DevOps. You see this is a snippet of the H T V P request, kind of like what you'd get from a das tool, but you wouldn't get this other stuff from the das tool. That data flowed through this application.

It ended up in this query in this method on this line of code. And we traced it all the way into this hibernate query. And we detected that the proper controls weren't, weren't used on this path, it wasn't escaped or parameterized in the right way.

So we can instantly warm the warn the developer with all the details here. Of course, under the hood we've got details about, you know, the full stack trace and all the lines of code and, and all the details here. I won't go into that too much, uh, for in the interest of time.

But I do wanna show you that we analyzed all the libraries here. We can also show you exactly how these libraries were used so you can focus in on the libraries that actually make a difference. And, and this is pretty cool.

This is, uh, that same application, but now we're going to, uh, and we can use it normally here, the application works just fine. You'd never know that it was protected by the stack hardening technique. But if I put in an exploit to exploit this vulnerability, we're safe.

The way that, uh, the stack hardening protects against that is it saw that request come in, I'll, I'll pull it up here. Saw that request come in, uh, looked at the attack and you can see it saw that the attack flowed through the application into the hibernate query and changed the meaning of that query. That's the definition of SQL injection.

That's what, uh, the instrumentation observed. And so that's what keeps our applications safe. So this is a really powerful approach to application security.

Uh, I'd love to talk to you more about it. You can reach out to me on LinkedIn or uh, or Twitter and I'd be happy to, you know, explain more about how this works or let you, uh, figure out a way to let you try it on your applications so that I'll stop. And, uh, thank you very much.

It was a pleasure talking to y'all.