Larry Maccherone – The 3 Ways of DevOps as the Keys to Developer-First Security

Has anyone here familiar with the theory of constraints show of hands? Okay, anyone want to try to explain it to the rest of us? John there you go, John you what's the theory of constraints John?

Exactly bottlenecks destroys. So so I like to use and I know some folks don't like this. I don't know if you're one of them John but I like to use the the weakest link analogy when I talk about the theory of constraints.

So if you strengthen Any link, but the weakest link. It won't strengthen the chain. And so the idea the theory of constraints is that you take that concept and you apply it to any value stream any flow of work from one step to another step to another step which is essentially all the work that we all do follows that General pattern.

If you improve that process somewhere other than the bottleneck you're basically wasting your time. So that's the idea of flow in the devops concept according to Gene Kim. So that's just an idea.

I'm planning in your head. We're going to go into that a little bit here in a little bit. But before that I did that I wanted to give you a little bit about my background and I won't give you the whole Spiel of whatever logo up there means there's a couple of things I want you to know about myself first of all, I'm a developer.

I write code every day. I'm the primary author of a dozen open source projects one of which gets a million downloads a month. I practice what I preach.

I'm I set up every project in the way that I'm about to describe here and I do it with the other contributors to these open source projects or if I'm on leading a development team like I did at Comcast. I I set it up the exact same way. I'm about to describe here as well.

So the other thing I'd like you to know about me is I mentioned Comcast I launched in scale the devset Ops transformation program at Comcast over the course of five years. I just so happen to have my contrast for my Comcast friends in the audience here and Sandra speaking tomorrow. I'm looking forward to that talk where she'll I'll learn how the program has evolved in the last year since I left but very capable hands very happy to learn.

More about that the concepts of flow feedback. And experimentation and learning which of the three ways of devops were the foundation for that program I built at Comcast and that's the topic of the talk here today. So the phrase SecOps gets bandied about it to be somewhat similar to this application security developer first application security developer-centric shift left.

Those are all phrases. I don't like to think of devsecopsis just devops lipstick put on a traditional security Pig. I have this someone new definition different than I had before.

Of of SecOps AKA developer-centric appsec and it's all about empowering engineering teams to take ownership of the security of their products we do this. In the devops three ways, and we never forget that the primary job of all software development is to deliver value to your customers in the form of software. And so we don't want to do anything that actually makes it harder to deliver value.

And in fact, if you do what we're about to talk about correctly, you will get both speed improvements as well as quality improvements as well as security improvements. They are not trade-offs you get an and of all three of those things if you do it this way. Um, so this is a picture of Gene Kim the I refer to him as the Godfather of of devopsy isn't the founder of devops, but he's got a lot of the writing that a lot of people refer to and a couple of quotes of his that we're going to actually pull up and the three ways of devops listed here that we're going to talk about.

So the first one of those ways is flow and so so flow I think is actually the key one and I I think that's probably why it's first by the way, None of these Concepts were new to be when I first read about them from the devops perspective. A lot of these come from the agile world and I think in particular flow comes from the kanban movement sub-movement inside of the agile movement. I think a lot of the concepts about theory of constraints and and the concept of flow really derive from from that.

Sort of provocative Lessons Learned this is sort of the the start with the end in mind and then I'll build up to why I think these are our good lessons and and why I'll try to convince you that you should adopt these as as best practices for yourself. So the first is this idea that you Center everything around this workflow thing that software development software developers are using almost completely for everything. They do nowadays called the pull request.

So most people think of the pull request are aware of the pull request as at least a step where you get a chance to do peer review of your code, but what a lot of people don't realize especially in the security world is that you can Add any number of checks to the pull request and these checks provide feedback at the right moment in time for the developer to stay in flow. If you're doing it while they're coding and they're trying to get the functionality to work and you provide security feedback. It'll take them out of flow.

It'll take them out of the mindset of get the functionality working. But after they make a commit and they like they update the pull requests and they want to see if it passes all the security the security automated testing. That's the right moment to provide that feedback.

It's also the right moment to provide quality feedback in the form of functional testing or unit, testing or any kind of test Suites or any other sort of automated check you want to put in that's the right moment to do that because they're not in the moment of trying to make the application do it with a new feature that they're trying to work on but they are in the moment of getting it ready for their peers to take a look at it. So the pull request mechanism is the mechanism by which a developer asks for that and so providing feedback right there at that point is sort of the key key to doing this the other and this is even more provocative and it's a little bit hyperbolic because I don't absolutely mean this all the time but Any service level agreement SLA. other than one day is not good.

It may be even harmful to not have it and I'll explain a little more what I mean by that in a little bit with some examples. Why is the pull request? The place for the developer.

Why does it keep them in flow? Well, there's also a psychological motivation that causes developer to sort of want the feedback at this moment time. So Developers Like the code, they're proud of the code.

They wrote this morning. They want their peers to like it. Also.

That's what the pull request gets. It gets a peer review of their stuff if these automated checks say hey your code has this problem. They're going to sit up.

They're gonna pay attention they're going to care about that. The code is also very relevant in their mind. It's very recently in their mind.

They understand how it works. The feedback will will immediately make sense to them because they just wrote the code that morning and so they can quickly resolve it if you wait and you triage the vulnerabilities and it takes three weeks for that triage to occur and then it goes on to a product backlog and then that has to get scheduled into a Sprint backlog and it's two or three months later before the someone on the team some developer on the team gets to work on it. It's completely out of flow and they it might not even be the developer that that wrote the original code and if if it is he he or she might have forgotten what that code was.

and so that's that's why I say Mttr any SLA greater than a day is is actually harmful if you can't fix it. The day you wrote it. That's the quickest time you can fix it.

If you build up a pile of these things, it takes 10x more energy to fix it. That pile will just keep growing. You'll never get out of that mode.

Okay, so one quote from Gene Kim that I said we would come back to and this applies to flow is that improvements made anywhere besides the bottleneck or an illusion now, he's not saying theory of constraints or or weakest link, but that's essentially the same same thing. So where are the bottlenecks? in Application security, so I'm going to give you a binary Choice here and we'll do a show of hands.

First of all, so the first choice and don't raise your hand yet. I want to get both of them up here on the screen here before you do so you can think about which one is the bottleneck for you. If you could improve only one of these things which would it be the first of those is your ability to more completely find vulnerabilities in application code.

And the second one is your ability to more rapidly resolve them. So show of hands how many folks think that the if you could improve only one that the one they would want to improve is the ability to more completely detect them one person here. no votes two people Okay, how many folks think that if you can improve your ability to rapidly resolve them you could do this.

So overwhelming maybe 80 90 percent of the folks that did this I did this same poll on LinkedIn and it was two to one was the answer there and actually poked at interviewed talk to the folks that that answered and and got a lot more detail. I don't have time to go into that here, but I do have a bit of a write-up on it on LinkedIn about that. So the bottleneck is not finding vulnerabilities.

It's resolving them. Why then do we constantly look for a better tool? That will find more things.

Also, if this is true, I think we're rolling out. tools all wrong So here's how I think you should roll them out instead of the way. We normally roll them out.

We normally say hey, here's a project plan. Let's get this tool running on all 500 or 5,000 or 50,000 applications we have and then we'll get the feedback and well at least know how bad it is and then we'll start to work on resolution. We'll leave that for a later exercise though.

We got to get it scanning on all 50,000 AppSec as a starting point. No, that's why but not deep you have to go deep first. So first of all pick one team and one application from that one team pick one tool category software composition analysis for instance.

And turn the policy dial down. So low that the team is willing to commit to resolve all those vulnerabilities in one or two sprints. That policy dial might be just criticals or it might even be just a subset of the criticals found if there's a bunch of them.

Once you do that you put in place this Branch protection status check in the pull request that it first just says, hey, you've got these these criticals in your in your in your application. And then as soon as they get clean the first time you switch it from a notify feedback. Where it'll just basically be an orange check mark and it won't block the merge button to a required status check.

That's the terminology for for GitHub at least which means it will block the the merge of the pull request so that it doesn't get to the next highest level environment. Unless someone overrides it you can always override it for good business reasons. Also while you're doing that install a second status check and you always have two status checks until you reach full maturity and the second status check will be the feedback mechanism that isn't blocking but that just tells the team how many highs you have how many of the next policy dial setting you have so it might not be highs because that's more than they can finish in the next two or three Sprints.

So you're always just making sure the policy dial is just two or three one or two three Sprints ahead of the development team and you keep notching up the policy dial this way to the point where it's it's the juice is not worth the squeeze and you basically stop with that policy dial setting for that tool category for that app. Now in reality the tools and the the critical levels. So this is the recommended progression that I usually recommend but your context may be different.

So software composition analysis analysis for the code. You import s bombs. There's so many For this I I lose track of them all and I listed though.

All the ones I could think of there. So so this kind of tool the tool that will tell you you have a vulnerability in the code. You import not the code you wrote.

This is the bigger bang for the buck. Just resolve the criticals for that then. Oh, sorry.

Let me go back. Then go on to the highs for the code you import or you could move on to the criticals for the code you the third one there the criticals for the code you write but but in general, I think it's easier while you're working on SCA feedback to get to criticals in highs clean before you move on then criticals for the code you write and then highs for the code you you write and then you can think about mediums or other tool categories after that. But that is the progression.

If you do even that first one of those I think you're gonna address a huge percentage of your risk, maybe 40% of your overall risk just doing the first one of those so you get value complete value out of it. If you spent all your energy spreading the tool across 5,000 applications, you get Negative value of actually running the scanning tool you only get value once you start to resolve them. So this focuses you very narrowly and focus you getting to Rapid most highest value you can get with the quickest.

Amount of time and and the least amount of effort that's the idea here. So there's some tips to pulling this off and I don't have time to go into into them too deeply so I'm just going to move across these and leave these for you for later because we're still only on on the the first of the three ways of of devops, but this is these are some tips that are that are necessary to pull this off one of the key things to pulling it off. And again, I'm going to skip past this slide pretty quickly because we have to move on but This is what I spend most of my time now talking to organizations about I launched a SecOps transformation program for a company over the last three weeks.

They have about 100 development teams. So not as big as the program I launched at at Comcast and we sat down with the most respected engineers in the organization. Not the folks that were volunteers for the champion program the security Champion program, but the folks that actually the other Engineers wish they were as as good at Engineers for we sat down with them and we sat down with the security leadership and we asked them to come up with a waiting of the things that are most important to work on.

Do you know what the number one thing? It was 55% of all the waiting that they came up with for the organization to work on. We need to be able to run our pipelines in parallel.

We need ephemeral infrastructure. We need container-based build systems. That was what they said was 55% of all of the value has nothing to do directly with security, but you can't do most of what else I'm talking about without that.

So it's very critical that you get that. Devops feedback way number two. And so I actually did talk a lot about feedback in the first section here that was labeled flow.

So I you think of that first section is flow and feedback together. So I'm gonna make this section be focused on metrics and and there's three purposes for metrics the first one and the one that folks most usually focus on is that second one. I have listed there to drive Improvement.

You give feedback in the form of metrics. So that folks know how good they are. It's like a Fitbit, right?

You know, you have the metric that says how many steps and then, you know, you can improve that over time. So that's the Obvious purpose for metrics, but the first one up there I think is is just as important when you're trying to get a program to scale across an organization at Comcast if we couldn't show that we had more effective for Less costs with the data and the feedback we were getting we wouldn't have gotten the go-ahead to expand to the next round of development teams. So I think this is really important that you capture metrics just to get gain executive sponsorship.

And then the last one and this is the topic of my talk at RSA. The last time we were alive here in 2020 was essentially to figure out which practices you think are good are actually good and which practices you think are good or actually not so good or not worth the not worth the effort and and so the feedback on the program itself, which things you're asking people to adopt that is the the third goal for that. So I have a visualization that I use to drive.

Feedback for development teams this cumulative flow diagram and at the top there you see it the feedback from an unhealthy team cumulative flow diagram. The top line goes up. When new vulnerabilities are found the green line goes up when they're resolved and then the blue line goes up when they're marked false positives.

And so what most people do is they track this green line. We're resolving a lot of vulnerabilities. We're making progress.

We're doing good. Well reality is is that the gap between the orange line and the green line is bigger here than it is here and that it is here. This is diverging.

The only reason it converged here a little bit is because they marked a bunch of stuff false positives because they knew they were giving them this feedback and that was the reaction. It's let's game the system and and Mark a bunch of stuff is false positives this other team on the other hand the top line goes up and the bottom line gets steeper. The green line gets steeper than the Top Line, you know that they're closing the Gap and then the other thing that's really nice about this is that the Top Line essentially levels off that means that the feedback has been so effective that they've learned how not to inject new vulnerabilities going forward and we're going to talk about that a little more deeply here in a second here, but that's a great slope of that line is one of the metrics I capture the Delta between these two lines normalized is another one that I capture and the area of this Orange.

A band here just going back a 90-day window is also something that that you can capture and drive drive metrics look another interesting thing about this. Positive line is that when a team marked these as false positives they were really false positives and we had to go in and tweak the tool. And once we did that this huge group of vulnerabilities disappeared from the open system because basically the tool now didn't recognize those as vulnerabilities because we fixed the the tools so the rule was actually capturing the right sanitizer for them.

So this provided feedback to the to my to my team at Comcast essentially to tell them we needed to jump in and tweak the tool rules there. So these were visualizations from from my time at Comcast. So when we use these sit down and talk with the team give them feedback with that.

This was feedback to the organization. So this is the oh, sorry wrong button. This is the primary visualization.

We used to track the overall progress of the program for a subset of the organization. So this was a 10 team organization. And each one of these arcs is a indication of How Deeply the team has adopted that practice this dark green color means they've adopted it all the way to the cultural level.

So a quick glance, you can see how green this is and you can see how well you're doing and then you can look at it over time and you can see how well you're doing you can use this to get to speak to an executive so that they can then or business unit leader so they can then motivate their teams appropriately inside of their part of the organization. I have open source this this visualization and it's part of a open source effort that I'm going to talk about here in a minute as well. Here's the two metrics I talked about I extract from the cumulative flow diagram.

I'll leave these for your reading for later as well. And I will point you the talk here I gave at in 2020 as well as a reference where you can see sort of the feedback on the program and how we improve the program based on on this data. Third Way for devops is experimentation and learning.

So improving daily work. Is even more important than doing daily work is the gene Kim quote here. Think about that for a second.

It's less important. To do your daily work than it is to do things that will improve your ability to do your daily work in the future. That learning effect is so much more powerful.

And that's what he's saying here is that experimentation and learning is how you improve over time and you have to focus on it. You can't say every feedback from a tool is just a checkbox. I need to check and as soon as I fix that vulnerability, I'm done.

No, you have to think of every feedback from every tool every incident production as a learning opportunity and you have to focus on preventing the next one of those that's what's meant by experimentation and learning. So there's a gentleman by the name of Watts Humphrey who is a mentor of mine when I was a Carnegie Mellon working on my PhD who's actually on my PhD committee. I use the founder of the capability maturity model.

If you're familiar with that which later went on to become the cmmi. It's more of you are probably familiar with that. But yeah, this thing called the defect three step and I've adapted it to be the defect incident vulnerability three step.

And basically it's what I just said don't just fix the things that were found find and fix others like it and do something. That prevent them from ever happening again in the future. So I'm almost at a time.

So I'm going to I'm going to leave a couple of these with you to read later. The Hedgehog of contextual learning contextual learning is when these three conditions are true. And the pull request is the best place to do this learning.

So contrast the company. I now work for we're partnered with a company named secure code Warrior hosting events together. They're actually sharing boost space on the Expo floor.

They do this perfectly when you get feedback on a vulnerability it takes you right to the Learning lesson for that kind of vulnerability and that's what contextual learning is Cold's learning cycle is also important. So I'll leave you with this last little idea and and I don't know if we'll have time for questions or not. But but we'll move on and I'll be available for questions after that.

We'll have time for questions. Okay, okay. so everybody likes to talk about well, what are the best practices for epsac?

What is the owasp list of best practices? What's the nist list? What's the the PCI framework?

What is it? Say, we should be doing everybody spends a ton of energy talking about these things and I contend that's not nearly as important as coming up with a way to get people to adopt any set of good practices. And if you use this feedback mechanism, they'll eventually be the perfect set the best set.

And so how do you do that? Well, you need a framework that takes development teams from wherever they are and takes them to very high level of maturity incrementally with coaching and a framework over time and that's what I'm working on as my side project right now. It's called transformation dot Dev blueprint.

There is a website up kind of in stealth mode. I've got a one company using it now and hopefully we'll open it up to Beta soon, but it's essentially a rewrite. Of the tooling we created at Comcast for doing this for managing the coaching organization.

I've got one coach in the in the in the room here from Comcast that I've got. I've got the the leader of of the coaches and the boss of the leader of the coaches here. So I've got I've got the whole the whole group here.

So all the ideas from that are embedded here in this in this thing, so, please sign up for access to the beta read the FAQ there. Let me know what you think. Thank you.