Larry Maccherone – How Your Cloud-Native Security Forecasts Impact Decision-Making

you Hi, my name is Larry matroni, and and I'm here to talk to you about this little idea. I have around considering every decision that you make as a forecast. And and we're going to talk about this in the context of cloud security decisions that you're making.

So this is a relevant to cloud cloud native development content. So we'll dig right in here with the idea here. So every decision is is a forecast is a guiding principle.

The idea here is that you are forecasting that the alternative you choose the choice you make will have better outcomes than the other Alternatives that you could have chosen. And so if if you actually think about it this way, normally you don't think about it this way. Normally you just like well that didn't work.

Well last time so we're not going to do that or even more common is probably this is the way we've always done it. So this is the way we're gonna do it. And there's not a lot of thought that goes into it.

We're humans. And and we are we make decisions a lot of times based on habit and patterns and less based on logic and analytic analysis and when the world changes that means that your habits and patterns from before are sometimes not the best decisions for the new world and Cloud native is a big change in the world. So this this talk is largely a discussion of how the assumptions that you have before that led to good decisions, maybe for application security pre-cloud native how those have fallen down in in the new world.

And and I've got one more concept. I want to introduce to sort of set the whole thing up and and that is from Edwards Deming and and he's not the only person that has a quote almost exactly like this, but I'm a dumb numbers guys. So the Edwards Devin version is the one that I gravitate towards.

So all models are wrong. Some models are useful and the idea here is is that you have to make models in order to make Forecasts and so you you can't make a perfect model ever, but sometimes they're useful and and the usefulness sort of disappears when they're so wrong that they make wrong predictions. And so you see I have to sort of find that right balance and we're gonna talk about essentially the assumptions that you are making right now or or maybe you aren't making them but but some folks in security World in particular would have made them pre-cloud native and maybe you you're not making some of these now but but we're gonna talk about them.

Anyway, just make sure that your models are are not baked in these assumptions are not baked in to your existing models before we dig into all of that. I want to give you a little bit more about my background than maybe the introduction so The the icons have sort of my history are going to come up on the screen here, but there's really only a couple of things. I really want to sort of focus on here.

The first is that for five years. I launched and led the devsecops transformation program at Comcast large organization with 600 development teams, 10,000 developers. I I basically replace the traditional way of doing appsec with this developer Centric approach to doing application security.

The other thing I want you to know the second thing. I like you know about myself is that I'm an active developer. I'm and the primary author of a dozen open source projects one of which gets a million downloads a month.

And and so I'm not just a theoretical preacher about Cloud native or security. I'm practicing what I preach I'm running projects continuously and I'm using all of these concepts with those teams that I'm putting together to build those those projects. So let's dig in to the Assumption.

So the first assumption is that cloud native is just a change in where you host your applications. And and and that's that's really not not valid it the the movement to Cloud native is motivated by a paradigm shift essentially where the targeted goal is to get both velocity and quality as well as scalability and reduced waste and so so it's it's a fundamental shift in in just the the motivation the reason that we're doing it and the other fundamental shift is in the architectures and these architectures. Are completely different in a lot of ways than what you're used to tiered layered architectures three-tier architecture is sort of like the sort of canonical architecture.

We had pre-cloud native and and you still have that you still could do that cloud native, but more architectural possibilities are coming up and more are actually beneficial get you to these goals of velocity scalability and reduce waste different architectures more effectively get get you to those and they're only enabled because of cloud native and and the general trend is tour. It's more Smaller and more disposable components pieces over time. And so you've heard this phrase cattle not prep pets.

So I brought up the picture of cattle here and it it's essentially superseding the picture of pets here. And so the idea here is that in the in the pre-cloud native world, you would take your pet to the vet and the vet would get fixed and the the pet would have a name and and you really care about the individual and and this is the way you treat your servers. You patch them you you worry about their health you you make sure they stay healthy Etc, you know in a container based world.

You don't really worry so much about individual continues. Maybe you have some monitoring there just like maybe in a cattle you might still get, you know, an individual cow treat it if it was if it was sick because the things are expensive right 1500 dollars for a head of a head of cattle in for certain breeds. And but but you care less about you don't name them and And you care less about the individual than you do about about the whole whole herd of them.

And then somebody else not too long ago, maybe a couple years ago said even cattle is to big and non-disposable chickens are much more disposable. Let's say chickens not cattle and and we're maybe we're getting to the point where we're taking the analogy maybe a little too far. But but you know this kind of fun.

So let's take it one more step even even further. But before that I would say chickens more represents the move to Technologies like fargate like AWS fargate where it's managed for you and and you don't actually even recognize the herd. It's almost like renting a herd kind of kind of thing or multi-tenant situations.

The last definitely taking the analogy too far is microbiomes not chicken. So the the idea here is that you really you can't even identify the individuals in the in the system. We're talking about like serverless functions those instances come and go before you you even notice them.

You're not, you know, you never really monitoring it an individual instance of a function call. But you do care about the environment that produces those that stands up those functions. Well, you do care about the species of the of the bacteria in the biome.

You do care about the mix of them you and you do care you do you do sort of change the makeup of the current biome based upon what you feed the biome based on the traffic you give to your system whether whether whether or not you're using a function serverless function X or service function why or message but Z, it doesn't matter you basically it'll it'll scale up and provide whatever is needed for the food for the for the for the calls for the for the traffic that's being provided to it. So we definitely took the analogy too far, but hopefully it'll stick now don't care about individuals don't think about patching them think about essentially the environment you create that essentially stands these things up automatically in response to stimuli. Okay, so there is a perimeter is is most most security people have have sort of come around to this at this point that there really is no perimeter or maybe you've heard the phrase identity is the new perimeter.

And and that is that is definitely true zero trust pretty much everyone in the security world and and not has heard of the concept of zero trust at this point. So there was always a risk that someone could get past the perimeter in the past, but it was lesser risk maybe and so you could focus on just protecting the perimeter and and sort of live with the risk associated with it now with the the Components of your system being deployed to multiple different clouds and multiple different places needing to be accessible from lots of different places over the Internet. It becomes even even harder to sort of think about about them as even having a perimeter and identity machine identity becomes even more more important.

So there's still some holding on to this idea here. There's still there's still vpcs out out there and I think of vpcs is a stepping stone. You still might have some assets that your systems that you're deploying Cloud native that are on-prem that need to be accessible by the system's deploy to Cloud native and so vpcs do enable that and and I say that's a stepping stone.

But once you push all of your systems to Cloud native, You really you really don't need a VPC anymore. It's really more of a safety blanket kind of thing. And and you really you really should should consider it sort of a just a step to get you to a true cab Cloud native and and zero trust movement is sort of the way to go about doing this.

So most people have sort of gotten to this way of thinking about the perimeter and and firewalls being ineffective and and the premise and and having people VPN through the fireball to get inside the perimeter to get access to things. They didn't have before most people have gotten past this at least conceptually if not in practice yet, but we still have a long way to go and practice here, but this is maybe even a little bit more Nuance so you might think well, there's no Global perimeter but like there's no Insider outside the firewall, there's no vpning in but now perimeters are just more local but there still is a perimeter around a particular system. I would actually Argue that now, even the concept of a system or system of systems is sort of becoming not very useful because it's very hard to define the boundaries.

They're all so interconnected that you know, any one drawing of the system or a system of systems is just sort of like a partial view of the of the whole thing that may be useful to you. So so the idea around micro segmentation, I think it's all so a stepping stone and I'm willing to sort of be wrong on this. I I think this that there are still probably some cases where Microsoft segmentation might be a great long-term concept, but I do believe in a lot of cases.

The juice is not worth the squeeze that if you're doing a really good job of identity between systems and systems. Then you really don't get a whole hatchable out of a lot of extra value out of the PCS and the same risks of firewall rules getting out of date apply to your VPC configuration. Sorry your microsegmentation configuration getting at a date you might have needed to open up something on a private or protected rail to to a sister service and then you forgot sort of why you did that and then you opened up three or four of those and maybe the sister Services have disappeared if you don't really know and so you can't shut them down.

So you still have these open holes in cases and and you you end up sort of making the decision and the problem a little more local but it's still there and and it's not gone away. And and this is this is a maybe preventing you from making the full. Inset shift to considering identity really the key concept here that you have to really get right and even get right from a machine to machine perspective.

I was just hit up on LinkedIn Yesterday by a company. That's that's now got a system for doing multi-factor authentication for system to system connection. So so I I'm not ready to talk about that yet.

I don't know enough yet, but that's the sort of thing that I think is sort of the right trajectory here and not micro segmentation. So I I posted on LinkedIn asking for advice and input on on this talk and and I got this really smart response it and it's much longer than that from this woman. I I don't I don't even know but Brianna Leaf I had to give her credit for this that's she's summarized my point almost perfectly so that all relationships must be authenticated authorized or otherwise denied by default and and that's sort of very simply summarizes what you should be doing nowadays.

Everyone every API call no matter how deep in the stack you think it is. You still need to do this with every single one. Outdated assumption number four kill chain modeling is key.

So there's this this concept of kill chains that it's been around a long while and I know Shannon lights you formally into it. It was a big proponent of of this for application security and there's there's other Frameworks Lockheed Martin has their own framework and lucky Martin actually sort of emphasizes that the framework is intended for advanced persistent threats not for a general abstract program. And so I'm not willing to sort of say that's not true that the this kind of thinking this approach to thinking might still be valid for advanced percent persistent threats advance for certain specifics and threats essentially as a as a fancy way of saying State actors.

Well funded actors not script kiddies not people that are in it just to make a buck. But but folks like like North Korea or China or Russia that are doing it for for purposes and willing to throw a ton of energy and money actually achieving a breach so kill chain modeling might be key for for advanced course and threats, but I think for for General applic Security it it's no longer no longer valid and and I use this swiss cheese model to describe. The both the old strategy and the new strategy that I'm proposing.

So in the in the Swiss Cheese model you assume there's like these layers of Swiss cheese and a successful attack happens when the holes in the Swiss cheese line up and the attack can get through and so you you worry about sort of what assets or at the bottom of what stacks of shoes. So if this is an important assets got credit card numbers and that gets more more emphasis and you and then you worry about what holes at the top layer of being attacked and and when you have a high prevalence of both the asset value and attack and you can figure out possible Places with the holes line up in the Swiss cheese, then you want to block the attack chain kill the chain attack chain at the sort of closest to the attacker point which is usually that top layer there. And so this this way of thinking is is been around a long time.

And yeah, there's the attack framework. I forget I said a miter a miter offering but this this is a bunch of variations. It's sort of had the same idea here and and I think this this is not really the way app section should be thought of today.

I think the new strategy is much simpler and much more effective actually eliminate as many holes as you can and and shrink the holes that that are left and that is the strategy. It's it the the juice is not worth the squeeze for trying to figure out if the whole line up and only close the ones when they line up it's just too much work. It's actually a lot easier to just close the holes completely in every one of the holes than it is to figure out.

When it is the work to do the analysis is actually swamps the work to do the Remediation in in these cases. so those are the the first four, uh Concepts that I wanted to sort of sort of first four assumptions, I wanted to sort of highlight as as being outdated and I'm gonna I'm gonna sort of move to a fifth one here, but I want to set the fifth one up a little bit more and and I'll give you the fifth one and then I'll then I'll then I'll set it up even a little more after that. So so I have this idea of of push sense making down and in towards the Creator or the creation or the consumer or the developer or the engineer and push trusted Assurance up and out towards consult the consumer and so part of the problem with the Swiss Cheese model over here in this old strategy is that you have to make sense of how the system behaves and if you're a security specialist.

It's very hard to do that in complex systems and and the more more complex. They get Cloud native enables them to become more. A complex.

They're runtime behavior is very difficult to predict and so you really want to only care that the developers who are developing it have a good model for how that behaves and instead what you want the developers and the the process the development team uses to push Assurance essentially concepts of assurance or evidence of assurance up towards the security folks. So they feel comfortable that the process and the tooling and the implementation are robust without actually evaluating the output of that process or the other actual software itself because it's very hard to make sense if you're outside of the development team and so we we don't do this. We we tend to try to sort of do things where the recipient of of the raw information is supposed to make sense of it and s bomb sharing is sort of the new buzz in the supply chain world, right, you know get every vendor to produce the next bomb on their software and then Dip it with every shipment of the software.

But how is the person who receives that s bomb going to make sense of that? How are they going to sort of know how whether or not the the those dependencies are attackable or not? And and it gets a very complicated thing and I think kill chain strategies.

Are an example of this as well? It's very hard for the for the folks who are in the siled security group to make sense of how the system behaves in order to do that analysis. And so you really have to sort of trust the developers a little more with that than you would have been willing to do in the past.

You sort of stuck. The other the other key Insight I want to make here is that in order to make a forecast you have to have a model. Maybe it's just in your head about how all the relevant Parts work and and the outside data assumptions make make that so I'm just repeating what I've been saying before.

I'm just sort of drilling it home that these assumptions being wrong is sort of the key. So what is this fifth? Assumption that I think is is wrong.

So we've got the first four. This is my summary slide of all five about to bring up the fifth one Cloud native is just a change in where you host the applications. There is a perimeter the perimeters are now just a bit more local and killing attack chains is sort of the key way to go about doing application security.

The fifth one is that it's Securities job to protect the applications. And this is a big assumption that is gonna be very hard to get a lot of folks to sort of agree. There's no longer their job.

And so that's why I want to set it up a little bit more and now I'm gonna spend a little more time essentially trying to convince you that that this assumption is wrong nowadays and then I'm gonna spend the the rest of the time after that just saying well now that we know this last assumption is wrong, what are you actually do about it? And and so so there's this quote from Gene Kim that any Improvement made anywhere besides the bottle neck are an illusion. And I love this quote.

It's actually a restatement of the the concept of from the lean combine world of theory of constraints. So it's not it's not a direct translation of theory constraints. But it's it's Gene Kim's essentially way of distilling it down to it.

And then basically the idea here is this if you have a a chain, let's say You've heard of the weakest link concept for China. I'm guessing if you improve the strength of any Link in the chain that isn't the weakest link. You actually don't improve the overall strength of the chain.

And so the theory of constraints is the same idea is that is that in human processes will work goes from one place to another one mode to another to another to another to eventually produce something useful is that and at some point in there there's a bottleneck and if you make an improvement anywhere besides at that bottleneck, it's like improving the Winks that aren't the weakest link. They don't actually make the change stronger and that's basically the idea. So what is the bottleneck for application security and and maybe this is sort of a general assumption as well that's wrong not a cloud native assumption, but in general assumption about about application security, and I contend that a lot of security folks are at least behaving like they think the bottleneck is that they need to find vulnerabilities but in reality even Any Improvement we make to finding more vulnerabilities is not going to help if you already have a big pile of vulnerabilities, you're having trouble getting fixed getting resolved and so fixing vulnerabilities and every time I do a poll for this.

I it's anywhere from from you know, 70% to 90% folks admit that fixing vulnerabilities when you think about it for a second is actually more of a bottleneck than finding them. And so um developers do the fixing. So so the work has to shift towards the developers.

And what does that leave security doing? Are they left with just cajoling? A lot of them have sort of decided that they need to have a lot of cajoling when the the developers have to fix something that they've found and so they've gone into been built up these cajoling programs and and I contend that cajoling and gatekeeping and policy enforcement.

These are sort of sort of roles that Securities come then comfortable with in the past that have been outdated by devops and Cloud native and the agile movements and lean movement. I I really think that those those no longer scale the moon the speed of ofment has gotten so fast. They can't keep up and everybody sort of points to that for the reason why you need to shift left and you need to have developers security, but I contend there's actually more Insidious and and bigger problems with this that are not really ever talked about or mentioned controlling is very time-consuming work we had Roughly 40 people equivalent people and wasn't people it wasn't the 40 people dedicated job, but it might have been 80 people where part of their job was was doing this but I estimated we had about 40 people essentially doing cajoling work at Comcast before the transition and and these folks are doing a job.

That's that's very soul-crushing it there may essentially telling someone their babies ugly all day long and and you most people don't like to do that. So those are super high turnover rate in these roles that sort of had a big cajoling element to them and the people that don't turn over rapidly other ones that actually sort of like calling somebody else's baby ugly. And these aren't the most easy to get along with people and they're not you're not building a relationship between security and Engineering with these people being in this role and doing this job.

So policy enforcement is confrontational along similar lines and and it best what you what you get is you get a checkbox response. Just tell me the bare minimum I can do to get you to go away to get you to stop complaining about my baby being ugly and that's not usually not very effective. The other problem is that a lot of these these folks are working toward off of a set of of policies and the policies are we're Bill and written over years, maybe pre-cloud native years.

So they're often completely out of date with modern development and devops approaches to to development. So so the the folks doing these audits are know this a lot of times and so there's like well it we can ignore that one sort of we can we can just focus on this one and we and then the whole thing seems arbitrary if there's some policies I can ignore then why can't I ignore all of them? You said you sort of get the broken window effect.

But if they were to throw a full book at the the team it would be way overwhelming this 300 Page thing where most development teams are not doing one tenth of the things they're in there. That means you know, you know some somewhere north of 250 Pages were there you're not doing and and it's very depressing and and too overwhelming to actually move people off. So what are you doing instead?

We would have to move towards this idea of developer-centric application security and and there's a buzz word that sort of a very popular to describe this called Dead Set cops, and I I won't get into it now, but have sort of a love hate relationship with that term So I I this is my definition of devsecops or as I prefer to call it nowadays developer-centric application security. It's empowered engineering teams taking ownership of the security of the software that they're producing. And they're doing it using the three ways of devops flow feedback and a culture of experimentation and learning and while they're doing it.

They're never forgetting and neither are the security people and neither are anyone else ever gonna forget that we only get value out of software when it's of use to the people that are using it and and if we if we have no users who cares about security and so the first job of software development is to deliver value to our clients and so we can't ever forget that that's why SEC devops always sort of annoys me because it's sort of implies that security is even more important and know some people don't don't think of it that way but but the dev has to happen you have to have an application and you have to have users with data worth protecting in order to even even motivate the security aspect of the work here. So, how do you get to this? How do you get an entire group of 10, developers 600 different development teams to be worthy of the trust of ownership of the security of the products, so I had to develop a whole framework.

A borrowed from agile transformation work that I did before and devops transformation work that I did before but I developed the whole framework for how you do this. And it's it's highly effective to the point where we're now doing it for a lot of other large Enterprises as part of my job working for contrast security. dev.

dev blueprint or just transformation blueprint and the concepts of transformation bluepoint are that you first have to leverage rather than work against developer psychology and development team sociology. So hook the brain of developers and change the process such that the pride in engineering Excellence is what motivates them to do the right thing. You have to enable incremental Gap analysis.

Don't throw a big book at them and expect them to do anything with it. Just give them the one two or three things. They could focus on next and and make a huge impact with with those one two, or three things.

You have to provide a shallow on ramp for this. So so you don't throw the next set of 10 things at them you throw them the first one two, or three things at them and then the next set of set of things at them Etc and you make this be context to a very narrow group of developers. One two, Pizza team one Squad at a time and they all evolve all the all mature at different levels.

But over the course of an effort of doing this you can raise the entire maturity of a whole development organization, but you have to think of it as one development team at a time that you're doing this because if you try to do it at as a whole group and mandate things some some folks will fall behind and then then they're just in sort of a hiding mode at that point and and you have no way of getting them caught up. Get each individual development team on this Improvement path is what I've been talking about just recently and then you coaching gamify them along it have metrics have a leaderboard reward people for the most improved in the last 90 days that those sorts of things get people get people to actually make it fun. And then and then you make people want to be on this leaderboard and that creates a sense of viral adoption.

This is the sentence sort of the general idea of the whole approach that I'm proposing to get development teams to be worthy of being trusted with the ownership of the security the products they're developing. I've got some more specific guidance that I have time to go into here and I'm gonna drill down into that a little bit more. So the first step is you have to identify this set of practices and not only identify them but you have to Force rank them and even weight them, you know, this practice is is going to be the first one we want everyone to do because it's got the highest bang for the buck this best Improvement in reduction of security risk for the That needs to go into it and and so much so will maybe even give it a weight like a number of points out of a hundred that that we can use to gamify with.

So how do you come up with this? This prioritized ranked list? So one one approach would be to borrow practice from industry standards.

Well, this is this is very you should you should if your security expert you you have some awareness of these and you should maybe use them as a checklist. But but most of the time using them directly as the list you start with is actually a recipe for disaster. They they don't all say the same thing.

First of all, they they target security Specialists both in the terminology. They use any approach. They certainly don't do it in a devopsy way.

They don't do it in a cloud native way. And and so so that's that's gonna come through and and sort of be rejected by the development teams if they'll just specify adaptations necessary in order to make it be developer-centric now now some of them have come along way with this and and so the OS one in particular is is is actually pretty good at at talking to the developer nowadays, but it's Little too big it. And and it it is no way anyone would do all of the oh I US Standard for this is really several hundred pages long and and it's too overwhelming.

So you really need that prioritized list and and the the this and they're not prioritized. It doesn't say this is the most important one to do first. This is the second most important one and you really have to have that.

And then even considering all that the general case. Your organization is different. Your organization has different risks or worries.

Maybe there's some things you're already doing really well and so the risk is lower over there already. And so you really need to focus in different areas. And so you need to have a plan that's that's for your organization.

So instead what I propose you do is you pull together your most respected engineers in the organization and and not the not the CTO not the VP of engineering unless they're the most respected engineers and they're still writing code on a regular basis. But the developers that are working on the coolest projects the ones and Architects and the ones that that really really deeply understand how software is developed because they're still doing it today and the ones that everyone else every other developer in the organization wishes. They were like wishes they had those skills wishes.

They were working on those projects wishes. They had all those great tools. And so you get those three or three to five most respected engineers in the room with your security leadership and you I I can facilitate these but but the the transformation to have blueprint software is is meant to sort of guide the facilitation of these as well.

By the way transformation blueprint is not out yet for open Beta. It's in private Alpha, but you can sign up for the open beta and hopefully we get to that pretty soon. We still have a little more work to do and I'm very busy so I haven't worked on as much as I wish I I had between now but anyway, so Out transformation dead blueprint though.

You can use Post-it notes or the electronic equivalent. And that's how I do these sessions today is I do them do them with Post-it notes and the trick here is to make everyone sort of put a stake in the ground independently, so you don't get groupthink and you don't get people to just say yeah. Yeah.

What what Joe said you have to actually say something in private first with your own set of post notes before you even merch try to merge them and these conflicts in the way people started. They're thinking off. We're all the Insight comes out.

We're all the interesting information all the sharing that was not being done before because people were talking past each other with different terminology and and they didn't want to have a comfortable conflicting conversation all of those come out in a safe friendly environment when you host a workshop this way and then you use the the output of this Workshop is essentially this prioritized list of practices eight to 12 of them not 40 or 400 of them, you know over the years the Them at Comcast got to be about 40 practices, but we always started every development team with the same set of six and then expanded to the second set of 12 in the end or some graduation like that. We shifted the graduation over the years as well. You take this list and you coach individual development teams with that which brings me to the next thing that you have to do to sort of pull this off and that is coaching.

And so I hired a staff of coachers coaches. Sorry to my organization to do this and and these coaches were not Security Experts. If you came if your resume came to me with one of those acronyms that said you had some security certification.

We went to the bottom of the stack at least the first few coaches later. I I when the role was better to find I started to accept some of those resumes, but for the first one I was looking for more of a Ted lasso type coach and I probably should replace the picture with the Ted lasso picture if it's not copyrighted, but I'm a hokey. So I I you know, this is the best coach of all time Frank Beamer for Virginia Tech football.

So that's what he he's up there. And this is a open sourced photograph that I'm allowed to use which is all the photographs in my deck. I try to have an open source license for but anyway, I would hire these non-security experts into this coaching role.

These folks were like Ted lasso they were good not At European football which he was hired to be the coach of European football when he didn't never even played soccer. He was a coach of an American football team, but he knows about getting people to improve and he knows about getting people to change their behavior so they can work better together and and that is essentially the the type of person you want in this role and then they use this this Force ranked list of practices to coach them to and they'll pick up a lot of security expertise along the way but they're really more of facilitator and and sort of problem solver an obstacle remover and and they don't really that need to have deep knowledge in order to pull that off. They need to have more social skills than that and the progression these coaches essentially walk each practice through and so my maturity model so to speak it's not a list of practices in material Level Two a different more advanced set and material level three Etc.

That's I don't really care about the list of practice I do but I want you to develop your own and I'll Be with that but but you have to have your own list of practices for me my maturity models essentially when you picked your set of practices and prioritize them how you progress through the maturity of each individual practice from thoughts to words to actions to culture and and thoughts means you haven't put much thought into it words means you're talking about adopting practice, maybe making some plans maybe identifying some obstacles that would get in your way that you maybe have to do in advance actions means you're in your your in process of doing it, but you're getting some partial value out of it at this point. So actions of installing a scanning tool but is not is not it's not actions that's still words in my book until you actually resolve all the criticals out of it. Then you've made progress then you're getting value and then culture is you getting essentially all the value you could for every possible place and you wouldn't think about starting a new project without this this practice in place.

And so thoughts words actions culture and coach will essentially sit down with each individual to Pizza development team talk about individual apps that they're responsible for and each app might be at a different point in and you might you might choose to do a practice for one app, but not another app and essentially go through this list Sports ranked lists to practices. And as soon as you find two or three that the team is not at the culture level for yet. You stop the Gap analysis.

So we're not gonna talk about the the even the eight or twelve you've got in your full list. If if the first three they're not doing and yeah, you certainly not going to talk about the 40 in the mature list in the first meeting and or the 400 in the in the in the OS guidance. You just going to talk about the the two or three they could Implement in the next 90 days and then you help them over the next 90 days do this and you come back and you do another meeting with the team another workshop with the team a resync workshop and and you help them pick another set of two or three practices to do and over the course of about a year and a half teams can go from very low levels of software engineering and security maturity to very high levels and this essentially shifts the ownership to the development team.

And this is the way you get the development team the sense making gets pushed down to the development team and the security people are providing these coaches less so than evaluating the product of the work that's being done by the development team and and kajoling them to fix it based on the things that the security groups tooling and and activities find Um, so the biggest pushback I get to this is that coaching individual development teams. That doesn't scale. I have to do something more across the board.

I have 10,000 developers 600 development teams. There's no way I can get to all of them. I have to do something more more blanket.

I only got three people on my staff. By the way. I only had three people on my staff at Comcast line.

I started this program. Um, so so there's no way I can do that and and the math doesn't doesn't bear that out that this is actually very scalable. It's certainly a lot more efficient and scalable than the 40 or so equivalents at Comcast doing cajoling work because there's no controlling work the coach never takes ownership of the problem.

The team keeps the ownership of the problems. And and so when you do the math on this though, it scales very nicely. So each workshop with the development team is is an hour and a half of the initial one and 60 Minutes the following one.

So average is about an hour and a quarter so If you do 10 of these a week, that's 12 and a half hours that you know that leaves you 35 or so other hours in a week to do other things to do ad hot coaching to do obstacle removal the work with teams. And so in a given week, you can you can get to 10 10 teams and there's ten weeks in in a quarter that are available to you. There's really more than that, but with PTO and such let's call it 10 weeks that you can do this.

So 10 teams per week 10 weeks a hundred teams coach can can handle at a not stressful State Pace a hundred different development teams that Comcast we had sort of the max of about 90 I think and and sort of the norm was when they got up to about 70 or 80 or so. We started to look to hire another coach to spread the load a little bit more out amongst more people so scales very very nicely with this approach. We got a Forex Improvement in efficiency.

It was one fourth the cost people wise to actually do this work then. A Jewelers and the other folks that sort of this program replaced and we had a 1/6 as many vulnerabilities in production our teams to do this. So there's a 24x value.

You can sell this to your bosses. Even if you only have three people you might have to tell them. Well, you know, we're gonna stop doing some stuff we were doing so we can carve off half of those people to start doing doing this new way and and maybe things will get a little worse for a while.

But here's where we're going to we're gonna get this or you could give me a little bit of budget that you were gonna give to somebody else and and we'll grow the program that way and it will later retire the old the old way of doing it. So part of the other thing that code that that your organization will secure organization will do is you're gonna stop being cajolas and Gatekeepers. You're going to be coaches and you're going to be tool Smiths.

And so the idea around toolsomas is is basically these folks are building self-service tooling that the development teams can use to to do this work themselves that they don't need to come to a security person to get a pen test to to or to get a scan run. They that's baked into their Pipeline and the scan automatically runs and it provides them all the value that a previous pentest word. Now, you should keep your pen testing team around to find things like Access Control logic problems and they work on those Advanced things.

But the vast majority of pen test results are things that could have been found by an automated tool and so you really want to push those to be found by an automated tool the other thing that these pipeline Engineers are doing and this is sort of probably the bigger value of it, but but Is one that's hard to sort of sell your bosses and I never got my bosses at Comcast to sort of really admit that this was a big deal because they kept saying things that sort of tried to defeat this this extra value is that they don't they're a development team, right? They're writing code. They're writing plugins to cicd tools.

They're they're building automation. They're building whole applications. In fact, the application.

My team bill was called Security Central and it was it was perhaps the biggest application built by by the security group ever at Comcast and there there practicing what they preach. They're not just Security Experts, they're developers and and they we're trying out all the practices and tools on that development work. And and before we push it to the development teams, you don't know how many false starts we prevented with this approach is incredibly valuable to make sure that our our internal team was using everything that we were going to push out to to the the actual Client development teams, so that's pretty much the content I have for you here today.

What's next? So, you know, hit me up on LinkedIn. Please questions like you can you can ask for a demo of contrast with a place where I work.

We implements a lot of these ideas and I can give you help as well with launching your own abstract program. Thank you.