Larry Maccherone – The Top 5 Risks to Prioritize for API Security

Hi, my name is Larry matroni, and I'm here to talk to you today about the top five risks for API security. However, I gave that title and Abstract before I knew that I would be the lead-off speaker today. And before I saw who the rest of the speakers were and so after I had that new knowledge new learning I decided to sort of up level the talk.

I wanted a sort of meet the the letter the law but a better title for the talk now is probably dated driven decision-making for cyber security using as an example the top five risks for API security or at least a reasonable set of those top five risks. So I want to start off sort of set the stage a little bit with what I'm talking about when I say dated driven decision-making with some examples outside of the cybersecurity realm to sort of open up your minds a little bit. and so one of the ideas I want to hit on is that conventional wisdom folklore intuition anecdote your gut feeling Up is often in conflict with the analysis.

Now. I'm not saying you should always go with the analysis and and not go with those other things, but I actually think you should factor in those other things into your analysis and really go with the analysis in the end and just to illustrate this point that coaches don't go for it on fourth down nearly as much and they're following conventional wisdom. So what's on the right here is sort of the three things that can happen on fourth down.

You can either go for it. You can punt the ball. And or you can kick a field goal and the two factors that factor into this decision are where on the field you are.

That's this Dimension up here. And how far you have to go to get a first grid down, you know, if go one yard or two yards Etc. And so this green area right here is the area where coaches go for it on fourth down more most often and you know, this big Orange area is how often they pumped it and this field goal area is how often they attempt to take a field goal.

The New York Times however, did a statistical model using this concept that I won't I won't go into called wins contributed that. shows that you should go for it. 10 times more often this green area here is 10 times larger than this green here.

And you know this analysis could be flawed. Right? I mean, it could be off by a factor of 2x or if after 4X, but even if it's off by a factor of Forex, it's still better to go for it more often than coaches typically do and and so what I'm trying to do is to sort of say that you'll win games more often you'll win in your career more often you'll win for your team and your organization more often.

If you do at least a little bit of analysis trying to figure out what's going to work best for you. So that's my intro of the concept. Let me step back a little bit and give you a little bit more about my background.

And so the the sort of icons of my my career history are gonna come up here on the left, but there's a couple of things I really sort of want to focus on first of all prior to joining contrast security where I work now, I launched and scaled the devsecops transformation program at Comcast Comcast has 600 different development teams, 10,000 developers and over the course of five years. We scale that program to to where we essentially replace the traditional way of doing appsac with this developer first developer-centric model and I got to you know, 300 of the 600 teams before I left but I've gotten the budget the agreement to get to all 600 now, I hear they're at about 700 teams in the program at Comcast. The other thing I want you to know about my me is that I'm an active developer.

I'm the primary author of a dozen open source projects. They're all have to do with data and analytics and so this is a very data oriented talk. So you're gonna see sort of why I I gravitate towards that a little bit here today one of those.

One of those open source projects gets a million downloads a month. So big project. I run it like a development team with all the contributors.

And so I'm practicing what I preach. I'm not just speaking as an industry expert or a consultant. I actually have hands-on experience with doing a lot of the things that that I talk about.

dev blueprint that is essentially a framework for doing a lot of what I'm going to talk about today. So I I sort of tried to argue for for why you should do analysis, but let's let's let's add a couple other principles to to the to the talk here and then we're going to build on all three of these ideas throughout the rest of it of the talk. So Principle number one is that you have a duty to maximize your the risk reduction.

for the budget that you're given people cash licensing costs Etc. And this is this is different than just saying you have a duty to do good things and you you do have a duty to do good things but you if that's all you're doing you're failing in the duty to maximize the value your organization gets out of doing those good things that's principle. Number one principle number two is that every decision is a forecast you are essentially forecasting that the choice or choices you're making will have better outcomes then some other choice or set of choices you could be Um, and and you might you you generally don't think of your decision making this way you you do things like oh last time we did that it didn't work out.

So we're gonna do something different today or or conventional wisdom says you should do this and this and this this so we're gonna do this in this and this and this or some standard says we should do this. Um, but but that's not the that's not that's not not actually forecasting the outcome directly. And so I'm saying that you should be a little more deliberate about that.

And if you know, it's a forecast then you should you should try to do sort of some modeling to make that forecast be as robust as possible without going overboard. So you let's bring this back to API security now a little bit and then we'll go off the rails again and do some more data Concepts and then we'll come back to API security for a very concrete example, but but if you're if if you're thinking right now is oh API use is growing. I've got API security tool vendors hitting me up.

We need to buy an API security tool and then then you're not you're not actually complying with both of those principles that we just so the first principle is you have a duty to maximize the risk reduction value of the choice you make um, but what else could you be spending? That money on the size and API security tool. And every decision is forecast this principle number two what analysis have you actually done to confirm that buying an API security tool combined with the rest of your choices is going to maximize that risk reduction value for you for your organization.

So I I very quickly want to come back to this and mention that this concept of winds contributed is within New York Times used and and you can argue. It's thought it it is flaw they admit that it's got some limitations but it's very complicated even it's it's Nuance is as they could make it. So it'd be very hard to use that as an example to sort of describe how to model this to sort of inform decision-making.

So I'm going to use a simpler example using a board game now. I don't know if I can actually see the public chat, but if you're if you're have a chat window available to you, you know, if you recognize this and and you're you play Catan, please let me know, you know in the in the chat that you're familiar with this board game Catan here Settlers of catanas is awarded the the game of the year a few years back. Well, that's probably more like a couple decades back now.

And it's a very popular strategy game but it's got a very interesting underlying statistical modeling probability probabilistic modeling that's sort of baked into it. And I'm gonna sort of teach you how to maximize your chances of winning at Catan here. And if if you follow this, I this Concepts and you're a con player, you will generally be every other player who doesn't follow the same approach almost every time not exactly every time because the role of the dice can sort of throw that off but but almost every time if you follow this this model you so let's let's define some variables for this model.

So let's say the value of a given resource on the board this resource here is the resource of wood. I don't know if you can see my cursor actually. Yeah you can this is she this is Brick.

This is wheat this is or so there's a number of different resources. Let's let's say that some resources are more valuable than others. Value is determined by the scarcity of the resource and how often you need it to do things you want to do in the game.

How often you need to use that resource. And so sheep is generally the most readily available resource in in most in most board layouts and it's the least used Because unless you're using the seafarers extension where they need they need wool from the Sheep to make sales. You're generally don't need a lot of sheep to build buildings.

And so so she is sort of like the least valuable. Or on the other hand is usually the most scarce because there's only two tiles on the on the board this tile and this tile whereas there's four sheet tiles. And so the the or is generally much more valuable than than the other resources and the other ones are sort of sort of in the middle.

So if you were to sort of say assign sheep a value of one and assign or a value 3 and give the rest of the value of two, that'd be a good approximation, but these tiles get get a number placed on each of them and and that number can drastically change the the scarcity of the resource. So an eight is a very high probability dice roll. And so that's on sheep that's gonna make sheep even more plentiful and so less valuable but this but let's say this only had ones or twos, um, or elevens are twelves on every sheep tile.

There's a 12 on this cheap towel here. That's not gonna come up as of So you could adjust that. One, two, three value based on the way the board laid out.

So first set that value then you you get to decide where to place your pieces on the intersection of three tiles. And this probability that that wood will come up and the value of wood time about times the value of wood plus the probability that brick will come up this for that just guys three dots times the value of brick plus the problem the value of sheep that you've assigned one plus the probability. It's got five dots that's the correlates with the probability the number of dots on these tiles.

If you were to add up all three of those sort of terms, you know V1 P1 V2 P2 V3 P3, that's the value of this spot on the board. And if those are the spots where you place your pieces you're likely to win the game more often. So so this is not a talk about how to win at Catan or football but I I did think it was important to give you some simple models outside of the domain so that you sort of understand what we're gonna do in the Messier world.

Of reality games have sort of very set structured rules reality is sort of harder to model. And and so we're going to come up with a simple model here that I think has actually highly effective. I've used it a lot.

I use it in pretty much all of the workshops that I do when I launch a desec Ops transformation program for a particular context. I don't use my historical knowledge of value and and risk to to do it. I use the knowledge of the people in the room for the for the workshop and I will challenge them if I think they sort of are are not seeing something but but we're gonna do the same thing here now for API security.

So let's set up the model for evaluating the value of a particular choice around API security. So R equals the risk addressed by a particular mitigation e is the efficacy of that mitigation against a particular risk and see is the cost the licensing costs are sort of hard costs, but the effort to actually implement the tool or the practice. It's usually a much bigger cost.

And so I sometimes I just say ether over efficacy is sort of the one of the terms of the formula about to give you And but but licensing cost does factor in so so this is a little more a little more nuanced. So the value of any mitigation is the risk that that mitigation is going to address because mitigating risk is is how you get value right times the ethics you see of that mitigation divided by the cost of that mitigation. So efficacy over effort is my simplified, you know bang for the buck kind of ethically for effort is sort of the this term times risk and that sort of simplifies to risk time efficacy divided by cost and so a mitigation might address multiple risks to varying degrees.

So it's often a Psalm of several value terms. And so this is sort of the more complicated expanded form summation of the model that we're going to use. So let's drill down to each of these terms a little bit cost is the most complicated one as I've already mentioned.

It's got licensing costs. Um bundles can complicate how you factor in cost. So if you if you if you buy in one tool from one vendor and a different tool and another vendor and a different tool from another vendor and they're all best to breed.

You might pay three times as much as if you bought or one, you know, two times as much as if you bought a suite from that first vendor and sort of lived with lower efficacy for the two other the two other tools. They aren't the best of breed but they might be sort of still to your one for that. So bundling dramatically impacts this so you have to model cost a little bit a little bit Nuance here effort to implement.

Of course, the initial installation set up is is effort but you know, that's a one-time cost. Um you but but the the effort to onboard everything or everybody is often a much bigger cost and a much bigger effort that you have to factor in. Um, if the tool is just going to find problems but not fix them or protect you from those problems.

Then you need to get Engineers developers to modify the code to remediate the findings. This can swamp all of the other costs often. That's the biggest cost but it comes out of somebody else's budget.

So that's something hard. Have you have to figure factor in so really if you're if you're just considering your budget you really only have to consider the the cost it would take for you to convince the engineers to actually remediate which is still a pretty big job that a lot of organizations the staff of cajolers at Comcast was up to about 40 people at one point. We got rid of most of that when you know, we transition to this ownership a model but the cajole or team they want to call the cajole or team, but that's essentially what they did all day long was was a big big expense that came out of security budget.

addition to the expense on the out of the engineer budget the time it took to remediate you have to consider cost sharing if you have an is tool for your web apps, but it can also mitigate custom code vulnerabilities in your apis. You can think of this as sort of a form of cost sharing. So if your model doesn't expand to include the sort of web app risks then if you have a simpler model like we're going to do here that's just focused on API security then you can sort of take into account that you have this stuff not in considered by lowering the cost attributable to just the API security risk aspect.

Um, these can also have complex interactions with efficacy. For example, it's very low effort to implement a wath in in record. Only mode detect only mode.

But if you actually want to turn it into blocking mode, it's a much bigger effort because you have to confirm that the that blocking will will basically block legitimate traffic. So this can get complicated you could either sort of model this as low effort and low efficacy or you can model it as high effort and medium efficacy, you know, depending upon sort of sort of what what you're going to do with the tool. Um, I'll give you another example where efficacy and interacts with with cost.

It pre-prod vulnerability detection might have medium money High effort and low efficacy if acquired with a dedicated API security tool because it's sort of like an add-on capability to it API security tool. That's why the the low money the low licensing cost. Um, but it might have medium licensing costs High effort and high efficacy if provided by a top tier AST tool and the high effort correlates with it's going to find more things.

So you have more things to fix but that's more effective at addressing the risk of of custom vulnerabilities in your in your in your in your in your apis implementation of your apis. So cost is the most complicated one as it always is with either the rest of these. I've got a lot fewer notes, then we can move on to actually using the model with some concrete examples here.

So as we said cost and come and efficacy have a complex interaction efficacy might also be split across more than one mitigation. For example, if you if you have a rasp tool that might be high efficacy and if you if you only had a rascal and if you only had a laugh tool that might be medium efficacy, but if you have both a rasp tool and a wath the Wrath tools gonna provide most of the efficacy so you might still model that as a high high efficacy, but then you the only you only need to consider the incremental protection provided by the web tool which would mean that it would only be adding to lower. So these are nuances around efficacy.

There's really less nuances. I'm going to talk About around even though this could be a big discussion around risk here. But the probability of the attack times how much damage that particular attack would do or kind of attack would do is sort of the you multiply these two terms to get risk, but but there's sort of some very general sort of understandings rules of thumb around around risk that will do.

All right. So let's let's talk about particular risks and particular mitigations in and tools so a big a big risk associated with apis. Is that you don't Implement your identity management your authentication well, or you pick technology.

Poorly there or you implement it poorly. I mean, you know jwt's is sort of like the cat's meow everyone wants to use jwt's which are which are bearer tokens. Meaning that the person who has one has access and you can't easily revoke that access as long as the token is valid.

So you put short timeframes on it to mitigate that a little bit. Um, but but there's a hundred of different parameters you can configure in how you implement your jwts and and I've done deep analysis on a bunch of implementations and I would say three out of four of them are very poorly implemented such that you're you're actually much riskier using jwt's then you would have been doing sort of a server side maintain session approach to Identity and authentication. And then your Access Control logic is is critical you want to make sure people can get at the things they need to get at but not get it the things they're not supposed to get that and that's sort of Coding these problems are very difficult to detect it's not impossible to detect with tools.

At least the tools that I know of are not very effective at detecting them. And so you really need humans to do this to do this work in the form of threat modelers and Pen testers and and security Architects that do either or both of those or or you know, the titles are different but this humans involved human evaluation of the design human evaluation of the implementation. Um involved in mitigating these things so known vulnerabilities with an available patch or an available last signature.

So software composition analysis will tell you if your code is using a dependency that has a publicly reported vulnerability against this is a big risk, even for apis because there's code on behind the API and and the way that you can sort of inject your attack is through the API. And so so the same problems that web app security tools have been sort of talking about for years not all of them and there's different there's certainly very enough differences with apis, you know in a pure sense then web apps. Um, but there's a this particular one overlaps pretty largely with with apis.

Um, and so software composition ask tools. I highly effective at telling you that you have these and then humans have to sort of remove that upgrade to the latest and Web application firewalls if there's a known signature for a particular attack against a known vulnerability like the came out after the log for Shell attacks started happening but a few days or a few weeks later before you could and then maybe even later than that before you could actually block with a laugh because you had to confirm that the signature didn't didn't break. Your your valued users are effective against these known vulnerabilities with known attacks.

Now zero day is there are sort of a different different world. Um, so zero days, they're not going to be in the sca tools database right away any and once they get into the sca tools database, they're gonna it's gonna take a while for that to get reported to a team and it's gonna take a while for the team to to remediate so and it's already being attacked if it's a zero day. That's usually how those your days to get reported log for shell for instance against a log for Jay zero day.

Um now arasp is actually a runtime application security protection is actually able to protect against zero days. In fact contrast. I refer to our Rask solution as negative day protection for zero days because they're not using signatures to protect.

They're using behaviors. They're looking to see if data gets from an untrusted Source like user data provided to an API and gets to a dangerous sink like an SQL statement without going through proper sanitization and they're looking for these Behavior patterns and so our our rasp tool and I would I would guess most rest tools would have the ability to blocked the log for Shell attacks even before you updated. So just natively our tool just if you had it running on the app, you would have been protected against log for Shell to and I suspect some other arrest tools had that capability as well.

Um vulnerabilities and custom code SQL injections that sort of thing we're talking about fast and I asked to sort of the two technologies that most people talk about with this. Although Das is also sort of catches some of these things as well not very effectively and for high cost so so I sort of left it out of the argument for vulnerabilities and custom code here. Um, so I won't talk more about those poor encryption or no encryption headers dance tools actually do to Tech these I asked tools also detect these as well.

And so so you sort of have a choice there that's tools detect them fast and easily typically as well. So so they're effective and low cost essentially for for that particular but I asked to do it also low cost and highly effectively I list Shadow api's DDOS attacks and overuse by legit. Users sort of in a cluster of sort of sort of runtime e things Shadow apis is is sort of like are there apis out there?

You don't know about this is a big risk because because if you're if you don't know about the meow, can you protect them? How can you monitor them? How can you sort of help the development teams that are producing those apis know what they need to do to secure them properly.

DDOS attacks are also a big risk for at scale. I mean if somebody decides they don't like your political position, they may decide to sort of shut you down by overwhelming your servers which means you're legitimate users can't can't use them. So you really do need you really do need some some DDOS protection as well.

So Cloud Web application and API production is a tool category that Gartner and Foresters sort of adopted. And that's sort of one way especially effective against DDOS attacks because the network traffic never gets into your network, which makes it kind of nice but you could buy a dedicated you could add Implement DDOS tax with your firewall, but that means you have to ask to come in. So it's very hard to block and you see you have enough bandwidth to let the bad traffic get into your network get to the outside part of your firewall at least.

So you paying for that extra extra cost there. Um dedicated API security tools are great at finding Shadow apis, that's probably sort of their biggest Advantage there. There's other ways to find Shadow apis including these Cloud WAP tools have sort of less effective less good ways of doing that.

I'm using cloudflare as the sort of example, I know a lot about because I I deploy to cloudflare for most of my most of my own development projects and then API Gateway sort of our been around a long time. Reverse proxies sort of got upgraded to all be API gateways and these can do some Discovery stuff. They're not very effective at DDOS attacks, but you can use them for rate rate limiting for legitimate users pretty effectively.

So so these are a list of risks and a list of mitigations and how they sort of relate to each other a little bit. By the way, poor encryption or no encryption and header problems can also be blocked by rest. It's sort of I I tried to make it look like there's a one-to-one relationship with the set of things on this side, but it isn't the case.

There's a lot of sort of cross cross overlaps. If I if I get a chance to update this slide, I'm gonna add some sort of cross lines to it later on here. And this is not a complete list.

I I did a post on on LinkedIn asking for sort of all the input and this and and this is not everything that that people provided me with. Um, and and this is already too complicated for us to model here today. So I'm gonna I'm gonna quote Deming there's actually another person as a similar quote almost the exact same morning but not quite but I'm gonna use demings version of it because I'm a huge Deming fannies and numbers guy numbers guy, right all models are wrong.

Some models are useful. So let's simple fly this sort of Complicated story that isn't even all of the story over here down to something we can with it's still useful and I think it's actually useful in reality that you could actually use the model and just, you know, tweak the numbers in it for your contacts and be effective but it's at the very least. It's useful to illustrate sort of how you would go about building.

So let's simplify the the risks down to design errors. This is sort of the the access control logic kind of things dependency risks that lumps together sort of the the known vulnerabilities as well as zero days vulnerabilities and custom code Shadow apis. We left alone as well and DDOS attacks.

We left sort of Standalone, but I'm picking these as sort of the top five that I think you should sort of consider right now. And so I said this is gonna be about the top five risks, but but it's really an example if you have a different if you have a different set of top five risks the model on giving you you could plug those in and it would still it would still work. So let's set up sort of a A/B test here.

Let's say you have two strategies for sort of addressing these five risks strategy one involves you employing a group of experts that do pen testing and threat modeling and and it also involves you buying a dedicated API security tool this tool pretty much all of these tools have pre-prod vulnerability detection sort of light weight compared to Dedicated s AST tools and and maybe false positive so sort of higher costs to use Because it you basically have to sort of figure out which ones are false positives before, you know, in addition to you have to say essentially do most of the the same work for false positives. As you do for True positives. They all have sort of some levels of runtime protection maybe along the lines of a wath but I wouldn't call them wax.

So, you know laugh like sort of runtime protection and they all have API Discovery and they do this essentially better than any other approach to do API Discovery. So so there's sort of gonna win API Discovery race here, but the sort of you know, sort of other Alternatives that are good and maybe better for these other two pages, but we've got those covered to us certain degree. So we're gonna go with this.

So we're gonna say this is our strategy and then we use our firewall DDOS protection built in so that's the way we're gonna address the details protection. Strategy two has the same Top Line bullet around humans, but instead of buying a dedicated API security tool we're going to buy AST tool Suite like maybe contrast which happens to have sca protection. I asked and rest protection and and you might buy a different Suite that has the sort of a different set.

Although check marks has the exact same they don't they don't have rasp they have sorry SCA is and sast So they have a different set of things and and different vendors have a different tool Suite so I'm gonna use the example for contrast having these three plus a couple others we have in the in the suite and you can buy the suite and if you buy any three capabilities individually, it's a lot more expensive. It's about the same as buying too and and the to get the whole Suite it's about the cost of buying any two of those capabilities cloud and I'm gonna rely upon a cloudflare like solution a cloud WAP web application and API protection to get me my DDOS protection Superior DDOS reduction over the firewall one, but inferior protection inferior ability some ability to do API Discovery, but not nearly as good ability to do API Discovery. So how do we model these these two compared to each other?

Well, we're just going to compare them. So the things that are exactly the same in both Going to just sort of remove from the model right away. They cancel each other out.

So the human stuff at the top and then the design errors sort of disappear from the model once we do that. So a little bit of a warning I'm gonna put some numbers up now and they are they're for illustration purposes. I actually did a lot of thinking about them, you know, risk efficacy and cost and I will give you my logic you time permitting for as much of that as I can.

Um, but but the point is not my numbers the point is is to model this to make your own decision about what to do and and so take the model and change the numbers to what you think. They should be and see if the model still gives you the same answer gives you different you should always do sort of sensitivity testing. Anyway, you should randomly change some of the numbers if you double this number, does that change the decision?

Well, that means you need to have me more accurate with that number. But if you double this number, it doesn't change the decision. Then you probably can live with sort of a rough estimate for that.

Um, okay. So here is oops. I pulled up I pulled up both so I forgot to animate the second thing on the right.

I'm gonna pull them all about both up at the same time, but I'm gonna talk about them one at a time. So this is strategy one simplified and this is strategy to simplified. The risks are going to be the same because I have the same list of risk.

So the numbers nine nine seven four so I I don't I don't think I'm likely to get attacked with the dust. It's a lower risk. The probability is pretty low.

And the Damage could be severe but it might be short-lived. So it's it's lower than the others. I think the risk of Shadow apis is there and it's significant but it's not nearly as as for my context is not near my example context not nearly as as big of a risk as vulnerabilities in the custom code independ.

Log for Shell log for for Jay attacks for the dependency risks sort of come forward to mind as well. As a lot of the supply chain attacks talked about and then sort of the traditional at you know cwes around SQL injections and stuff. You know, we don't want any anyone attacking that stuff.

So I this is my rating for risks and it's the same over here. So the only differences are in efficacy and costs and so the dedicated API security choose have pretty lightweight protection for vulnerability detection pre-prod both SCA and vulnerabilities in the code you write and they don't have any protection for zero days. So so you basically I'm giving them a low efficacy for for both of these categories a three whereas these tools have sort of robustly been built around those problems and I'm giving them a pretty high.

Here for those but they're much more expensive because they're gonna find more things. So it's going to take more effort to actually address them. And then that's essentially why the cost numbers are much higher over here than they are over here as well.

Shadow apis, you know the the dedicated API security tools do that better than than sort of any other approach I've seen so they get a 10 the highest number I have on this board. Actually. It's a 10 for that and the risk is still pretty significant and because it's part of a bundle the cost is relatively low and there is some effort though to onboard and get things going through the API Gateway get things going through the the the dedicated API security tool and so so there is some some costs associated with most of the cost is that is that effort though?

Not the licensing costs because you're getting it as part of a bundle with a bunch of other thing risk is the same the efficacy of sort of a cloud for like solution in finding Shadow apis is a lot lower. It doesn't do skiing the detection except for open API. Whereas I know I know one vendors in the space will detect you know, GR.

The protocol buffers various flavors of that will graphql Etc. And so there's a lot more API robustness there. For that um and DDOS attacks low efficacy here because because you can never buy enough bandwidth coming in on the other side of your firewall.

You never want to pay up front for that much. So someone could always a swamp you and flood your network capability. If you do it on in your on your premise and then the cost is high because you have to pay for enough of a bandwidth on that side of the firewall to essentially make sure you can handle lower levels of DDOS attacks.

Whereas if you use a tool like cloudflare or service like cloudflare the traffic never gets onto your network. Um, and and that's sort of one of the things they do better than anybody else so they get a 10 for efficacy and they're sharing class across of maintaining that bandwidth across all of their customers. And so the cost is much lower for that.

And so if you just sort of use the formula are times e divided by C you get this and you get this and you get this and so it looks like this strategy would be Better strategy using these numbers then using this strategy. Now, here's where the sensitivity comes in if you decided that shadow apis was your biggest risk. The this would flip, you know, if you if this got a 10 and these were lower risks this would flip and this strategy would be with dominate because that's what they that's what they do better than essentially everyone else.

So you have to do some sensitivity testing and you have to sort of put your own numbers in to the model. So that's my model. Um, you know, I I am I want to come back to my Sharp at the beginning um, and ask sort of a little a little sort of human question about this this analysis.

So so if the analysis says you should go for it more often. Unfortunately. Why don't coaches do that.

So, you know think about that for a second and and you could type the answer. I can't see it directly as to get relayed to me. So it might not be quick enough before I sort of expose it.

But but um, I'll take a sip of my things so you can type your answer if you have one. Why don't they go for it more often, unfortunately? Contend that the reason they don't go off go go for it more often.

Unfortunately. One of the big reasons is actually not conventional wisdom is that I think conventional wisdom became that for what I'm about to say because of what I'm about to say that that If you try to punt the ball and it even if the other team runs it back for a touchdown that's not attributed as a bad decision to the coach. That's just poor execution of the team.

And so it's not directly attributed all the coach same thing with field goal. If you try to kick a field goal and you miss oh well, that was just bad luck on the field goal. Kick.

Oh good luck on the other teams part of blocking it for instance. Um, and so with that also is not attributable to the coach. But if you decide to go for it on fourth down and you don't make it on Monday after the Sunday football game the press all they're gonna be talking about is that bad decision to try to go for it on forth down is directly attributable coach.

So I think their career optimizing for their career is what's going on here and they're not actually optimizing for the coach winning. So we we do have some questions coming in. That's my that's my my slides and I only have three minutes left here.

So this one question is around risk how to rate it by yourself any references for an attendee and also cost. You know, why? Why did I do that modeling, you know, so I I do offer a workshop where I can help you figure this out the This sort of the thing that I want to sort of start off with saying though is that you probably have more insight into this then you realize and so my workshop really focuses Less on me giving you the answers or my Logic for the answers is really what matters.

Um, then me sort of getting the right people in your organization in the room and extracting the answers for your context and risk is very context specific. And so I can't give those in advance in general terms. I you have to really know what you're what you're what your situation is like before you can you can say with the risk is but I can pull that out.

I I host these workshops where I can pull that out and about two thirds of a day, you know, three or four meetings promote an hour an hour and a half each. I can pull out sort of you a model like this sort of for you to come up with your strategy for the next year for your cyber security movement your effort. And and that's what I do all day essentially.

That's that's my primary job at at contrast and and I do it as part of my side job as well with the transformation that Dev um kind of thing so so I can glad to help you with that but If you don't if you don't want to engage in my help, I'm you know, I would say pull together a group of people in a room and using this example, make sure they all sort of watch this video in advance and and you can set up sort of a whiteboard or a virtual whiteboard with sort of the table like I've shown and you can sort of talk about what numbers you should put in each of them and you'll get pretty close. And then if you do this sensitivity testing, you know, you'll get to the right place. So that's all the time.

I have everyone. So thank you and and I'll hand it back to Alan or whoever the next speaker is.