Development Security in a DevOps World | DevOps Onramp 2023

it Hi, my name is Don mcvitty. And welcome to development security. I'm excited to be speaking here at devops onramp 2023.

My background is in the breadth of it. I started Life as a developer. That's still my first love but I've done pretty much every role in it and even developed embedded code for smartphones so pretty much you name it.

I've done it. I've been an operations manager. I've been a development manager.

I've implemented devops and Security in different organizations. I've never actually held a security role interestingly enough in it as many of you know, someone has to do the security and not every work and afford to keep a security staff. So that's my security experiences because it needed to be done and I enjoy doing it so that helps a lot.

We're going to talk today about the different tools available to secure your application specifically while you're developing it. And that's why it's called development security. They used to call this devsecops but devsecops is actually wider and at the end of this we'll have a little bit of a discussion about adjacent tools that will help protect your application also, but don't fit into development security.

So first off the tried and true is static application security testing. It actually just scans your source code. Looks for vulnerabilities or bad coding increasingly bad coding practices also and notifies you of them.

It can run directly in your IDE. It can run from the command line or it could be run as part of the devops tool chain and often several of those will be used at once for What's called sast which is? static application security testing you run the scan you get the results the prioritized for the better tools and you go through the prioritization list and fix what you think needs to be fixed next to that is dast which is dynamic application security testing.

There are a lot of things that scanning the source code can't detect and scanning the source code requires a scanner. That's knowledgeable in your language. They asked tries to resolve both of those issues by scanning the running app.

or the completed app thus not needing the source code. and looking for places where on we're vulnerabilities are evident to the outside world. Um desk is run normally in the devops tool chain directly in the chain rather than in the IDE or something because it's the completed after worried about not the piece you're working on or the source code of it.

So it runs gives you a prioritized list of known vulnerabilities or things that can see right. I think probably the easiest one is buffer overflow. Probably got caught in.

In SAS scanning, but if it didn't for some reason because of the way the code comes together, you know accessing off the end of an array because you're adding two numbers and that's it's not obvious that you go past the end of the array until it's running that type of thing can be detected by dust. The problem with that was it didn't test the actual application in its actual World which in the security space is a big deal. So companies increasingly are implementing.

We interactive application security testing or I asked and the point of is to do everything gas does but also Look at the environment that the application is running in so it matters. If you have a certain on think of it like feature flag development, if you have a certain flag set in the environment the application May behave differently. Well, the point of I ask this to take all of that into consideration and look at the application as it will be deployed usually in a copy of production environment in some places directly in production.

And again, it gives you a prioritized list you can go through. and check everything off now the way that That I I asked works and dassed. Probably more the stuff we're going to talk about in a minute.

But but dast and I asked are the start of they're using vulnerability databases. So most people are familiar with CV even if they're not security people. It takes feeds from those databases.

That's just one of many and uses those to help test the code. And then the cve priority is already assigned. So it doesn't have to really do a lot of work to give your prioritized list it goes.

Well, this one is worse than that one because CV says so so here's your list pretty easy pretty straightforward. And makes the code more secure across the board. The problem is that if you detect.

At runtime say you're in production and you detected an error. You get a prioritized list of all the vulnerabilities that it found. now what most organizations are moving faster than they ever have but it's still a while to get a ticket created and get someone working on it and get the fix into place.

because of that delay then organization started working with something called grasp, which is runtime application security protection and point of grass And it is available in most of development security tools, but not all yet. I think it will be eventually the point of rasp is we found it. We found an error protect against the air in real time while we set up things up to get it fixed.

So on the phrase insecurity is shift left protect, right which is catch bugs earlier, but when you don't catch them protect the running system. and the idea of rasp is pretty simple, but it turns a lot of people a lot of organizations off because it puts a small amount of code in front of Every exposed point on the application now every exposed Point varies, but for sure everything exposed to the world, excuse me, and the point of that. Of that small bit of code is to keep whatever the vulnerability was from happening.

Basically a miniature firewall if you could you can think of it that way it works, okay. The reason that it upsets some organizations or some app teams. Is because it's not performant.

I mean, I'm not saying these these organizations are writing bad code I'm saying every time you hit that point extra code has to run and for time sensitive or performance sensitive software. That's not an acceptable solution. So you have to look at is this the right place to use rest, but if you're not really worried about performance It's a great place or a great tool to have in place because it's another protection.

And let's face it. We don't fix every bug right away. So having something there to let us sleep at night.

Probably a good idea. The one thing I would caution against is that Organizations once they have a protection in place regardless of it if it's rasp or a web application firewall or something else. Once they feel their protected the priority of fixing that bug goes down, but that's really not good.

Standard practice you really want to make sure that you know, okay. This vulnerability is in there. We're protected long enough to fix it.

Let's fix it right away. And of course prioritizations change the one that's not protected you want to fix first, but you don't want to ignore the ones that rest is covering for you right now. Just my word of caution.

For software build materials, which is the next one. This is completely so all of those were evolutionary. These steps came along right through from Fast to Grass.

They slowly built up better and better protections and it actually started before sass before Seth was was lint scanners for C and Python and pearl or seeing Pearl and PHP. Sorry. So this is this is actually an evolutionary path.

You can follow as bomb is different. So as we started using more and more open source software. We became more vulnerable to.

Major issues. Um Probably the best one was I don't remember the name of it. But when SSL?

When a developer was rushing out to Christmas vacation and accidentally committed bugs to SSL and everybody's sslope. Oh, we were all dependent on that piece of software and when it broke we had to go find everywhere was in the software. The same is true of a variety of other issues that have cropped up.

So there was the response to this was people started going. I need to know what's in my application what was included by me? And my developers and what was included by stuff we included so if you include a library and OSS library and it needs some lower level functionality, it may include five other OSS libraries and we needed to know everything what's in our application what makes it and that software bill of materials what's in it?

It's much like the bill of materials for manufacturing same type of thing. This has been around for a while and they've been making it slowly better and better but the United States government. By executive order a couple years ago said no, if you want to be a vendor for the US government, you must have the ability to generate software bill of materials so we can see it.

So we know where our vulnerabilities are. That made everyone and by everyone I don't just mean security companies or development security companies. I mean pretty much all of it all of those vendors started generating Xbox.

started providing tools to generate s-bombs There is an open source project that most of the tools out there are based on but really source code scanners already did this. So in a moment? We're gonna look at the next thing that builds that uses s bomb.

It doesn't build off of it uses S5. If they were doing the next thing the narrity had s bomb functionality inside their scanner tool. They just had to present it in a way you could use it.

And so there was some work. To pull out what they already knew and presented in a text file and a format you can use and slowly we've developed a couple of standards for how to present s-bombs and you can now just get software the open source project. I mentioned most development security tools.

You can just click a button and say generate an s-bomb poof off it goes and you get a list of all the software used to create your application. now these are Big lists. There's a lot of stuff crammed into any modern application.

You can have 20 lines of code and lying underneath it are 40 included libraries with tens of thousands of lines of code. So just the way modern application is developed or my moderate software is developed. So the ability to go through this s bomb intelligently is important and rather than have you pass through every s bomb that you generate for every application that you own and all of its inclusions.

They came up with Source composition analysis now, so as composition analysis was already going on inside security software, but now it's a separate thing and it basically looks at the s-bomb Compares it to the CV databases, which is earlier. I said more later well as Source composition analysis or SCA. Is dependent on those databases it needs to know what vulnerabilities are.

We aware of? and where are they and then it will scan through your your application find all of those references in the s-bomb. and tag them so if it turns out tomorrow that You know the standard C library s printf has a bug in it that potentially could like hackers in.

Everywhere that exists needs to be flagged. So you know what your vulnerability is your exposure is. These tools will do that.

And do it by saying okay. There's a cve in the data or there's a entry in the cve database that says, you know, that's printf has a memory over run. so everywhere there's s print off we need to tag it and then CV database also has a severity so we need to tag it and say how severe is and they do that for all of the included stuff in your library.

This is huge on. If you had to go through every one of those super complex s bombs by yourself. I think you'd struggle.

I think every organization would struggle Source composition analysis. Kind of shortens that path by only showing you the ones that have known vulnerabilities. Now, the vulnerability is range from this will never happen to oh my goodness.

You need to patch everything right now. And something like my example of us printf is a really bad one because it's not like you're gonna change. every aspirin def call they're gonna fix the library rather than you fix every call, but you need to know where You're exposed and between with the sca being able to take the s-bomb and compare it to vulnerability database entries.

It can tell you where You're vulnerable at least. So all of these taken together are the core of development security they help you while you're developing code to help you while you're building the code. They help you while the code is running they let you know what is in your software and they let you know what vulnerabilities are and things you included.

that you don't have control over so that you know if you have to swap one library for another just because the one you're currently using has too many vulnerabilities in it. It's a broad array of tools and they've come a long way. I mean the ability fast and even to a lesser extent dashed can actually tell you while you're writing code.

What you're doing wrong? security wise and that's astounding compared to what we've had in the past which was right, the entire application security tests it and then comes back with a list of things you need to change. Now while you're thinking about it while you're right there in the code the IDE can tell you.

Oh stop that that's vulnerable do something else or you know, the way you made this call, it could be vulnerable. You should consider changing it to this other way nice stuff really nice. We've come a long way but also means that security which is traditionally had a problem in devops now has An integrated role in the devops toolchain each of these fits in the tools in the tool chain does its job while the tool chain and all the process is going and out the other end of the devops stack comes a secure application big deal and kind of thrilled that we have them on.

It's not everything for security. Sadly. We I'm sure you know if you work in it at all that we're finding new.

Ways to address security issues every day because there are new security issues every day. So there are adjacent tools things. I like to talk about that you can use in addition to these development security tools that are similar and sometimes our implemented right in a development security tool with all these other features.

so the biggest one I think is API security development security looks at the code. API security right now looks at the interfaces between pieces of code and in API heavy development particularly and like microservices. You really need to run API security whether it's built into your development security tools or you buy a separate product.

You really need to or Implement an open source project. There are some you really need to look at the apis and understand them. Are they secure Is a big deal.

I'm not a fan of the current space where Das looks at the source code without concern for how it interfaces to everything else. API security looks at the interface without security concern for the code underneath and I want to unite solution, but that's coming. It's I'm asking for.

You know things that we didn't realize we needed until now, but not sure it will come. There will be a we look at the interface. We look at the code underneath we come with a complete list of issues that you can fix.

It's coming question is when but for now Implement API security whether you do it through an API security product and there are several good ones on the market or through a web application security or web application protection dependent on who's acronym you use Wes or web. Whether you do it through either one of those do consider API security. Along with that web app scanning.

It's not. so scanning the completed application is different than to ask their IAS on this scanning watches the application all the time to make sure that there aren't changes that are causing it to be vulnerable today. Um, sometimes I asked his implemented to do that web app scanning is focused specifically on web applications.

So it's a little bit better at the job. Even though I asked does borrow a lot from the space. So I use both in the short term in the long term.

I think that they will merge into one market. I just don't know what they'll call it. The other two big ones are zero day, which rasp takes care of.

For most things but you do want zero day protection in something like a lap or a wax where? It can detect the problem and then help you fix it because again just like with grasp you want you want to shift left protect, right? So once it's detected and it was never known before rasp was not great at fixing that because we're ask works on what's in vulnerability databases zero day tools are increasingly using AI but have used a lot of different methods to go.

Okay, that's not good behavior. Knock it off and block bad behavior. It's just another layer of protection you can Implement and they've come a long way so you don't have to worry so much about false positives.

I mean, all of these tools used to be false positives where they're Bane. now I think people are more worried about false negatives than they are false positives. Right?

Did you miss anything is the quality of the tool high enough for me and that's something that you can you know. Look at different evaluations of these tools and find out. As you implement, but the whole point for zero day is stop it before it gets to you.

And the only way to do that is with tools that are running all the time and watching traffic and analyzing it and increasingly AI is going these are normal behaviors. That's not So zero days a good addition to these and the last one would be DDOS. You can't do DDOS with any of DDOS protection.

Or even dos protection with any of these application or these tools the tools we've talked about for development security. Don't keep your app online. What keeps your app online is something on the network making sure that those attacks aren't opening connections, which usually means outside your network completely unless you have a ton of bandwidth, which we don't right so Usually it's a hosted service that just says Whoa, all these people are I'm taking a super simple example.

So bear with me all these people are opening connections and then doing nothing. And so that would bury a web server, but we're gonna block them because they've got a certain pattern we can detect sort of stop it. Add those protections zero day DDOS API security web app scanning to the list of tools that we've talked about today and you've got an impressive.

Amount of protection for your application. Let's face it you spend a lot of time developing. It spend a lot of time getting it running you're maintaining it all the time.

The last thing you need is Some person for whatever reason be it state-sponsored or some kid in his his mom's basement. Whatever. Attacking you and taking you down.

You don't want to lose your customer data and all of these tools are aimed at helping you protect your application and doing it easier than it's ever been able to be done before the ability. To get instant feedback from all of these or feedback at the right time at the right place from all of these is huge. You should be using them.

There's a lot more to them than I can cover in a in a short presentation. So you want to go out and find more information about each of them on most of these coming batches are big vendors on if you look for development security, you'll find the big vendors and they usually have all of these all of the main ones. I talked about dashed I asked for asked s bomb and SCA so between them.

At the a lot of protection for your application and if they come bundled you only have to install one application. To get all of that protection running now, of course, there are add-ons for IDs you have to install and such but it's all automated. So once you get one and install it You're done and from then on you can just use it to protect your applications.

That's my time. It's been a pleasure speaking and I hope you learned something. Please reach out by social media if you have any questions and or want to connect.

Enjoy the rest of devops on ramp 2023 and check for my blogs. They're on mediops. every week Thank you.