John Willis, Red Hat | DevOps Connect 2022

All right, what would Deming do in a double quotes cyber of the whole world? Who am I I mean this business 40 years. So I look like younger and I am but I I guess I'm most famous for co-authoring the devops handbook.

But I also do the labor of love with Jean Kim something called beyond the feety's project. And then over the last couple of years. I've been very interested.

In fact, I spoke at the first devops. This devops can act of secops. How long ago was that?

Mark was the first one? 15 or something like that or Later, okay. Well, I have to fix one of my slides and the I can't really interested but I was more interested is not a security person.

But like how audit And how people think about so I've worked on two projects one in 2019 called out so automated governance. And you know, it's time to timer. Just so I know and then the Peter over here.

Was if you want to talk about really interesting product this on a cloud and now it's called Cloud notification Cloud security notification framework, right? So it's it's really exciting stuff. I'm gonna waste too much time if I should explain it.

It's still my presentation, but it's it's the average in general. It's an aggregation of multiclouds to be a normalize information. It's an incredible project so you can start by going.

Oh no, but so in 2019, we wrote this paper on the left automated governance where we wanted to prove out that we could increase the efficacy of audit. in lower the toil of order because you know most large corporations 30 40 days a year. It's just Chaos, right.

It's it's subjective arguments. So we want to see if we could do an objective implementation of a sort of digitally signed and by the way, like no disrespect but s bombs are necessary but not even nearly sufficient. So if you want to talk to me about that, but so this is a really like focusing on audit and at the point if you figure the audit problem out a lot of stuff so last year we got back together.

We thought we do a version two. But we decided the first version was really boring because it really is a reference architecture and like even the most staunch like technologists would like all right enough the 60th at this station and gate like I can't take it anymore. So he decided to write it as sort of a follow-on to the Phoenix project.

So this we started as a novella called Investments unlimited where basically the it's a bang. It's Investment Bank smaller and and basically they get an Mr. Mr.

IA like which is the metaphor immediate requires immediate attention like like so it's it's basically failing an audit and to see you know gets it it's just it's the classic like CEOs. Like hey, I thought we were doing good with this stem-outs thing. Do you guys just speak at the big devops conference and like yeah, but not security.

And it is this story is basically written primarily about a bank that is implemented in automated governance structure that basically creates digitally signed at the stations in Gates all from with no human intervention. It's incredible and if you want to talk about it, the book should be out this summer. They turned into a book and then this presentation.

Unfortunately, it's not about that. It's about my love affair with this guy named Edward Deming and I basically have just finished a book was a 10 year label of 14 months. It's in the second draft.

I've done a lot of stuff. So like I saw the company Docker I saw the company Dell. I I don't I was early in at Chef.

I was going to co-founders devops days. I actually wanted to co-founders of the second stays with Mark Miller over there. Roughly speaking.

And today I work at Red Hat as a global transformation officer. I don't know what that means. Then as used a trick if you're gonna write a book do a podcast.

It creates so much fodder for you. So that's what I did about a year and a half ago. com has all the it's really some fascinating podcasts that I the different people like.

I mean just this is why I always run out of time but I interviewed a guy who was like one of the original autonomous vehicle guys at GM. And he started it in Motorsports like that whole angle like fascinating. Yeah.

I'm sort of obsessed with Dr. David. Somebody said that is at a younger me like that was pretty terrible.

But hey, we're gonna do so if you haven't heard it damning so you have heard of Demi like this the first off it's mandatory in every devout presentation quote. It's like I hope you didn't break that rule but learning is not closer. You neither survival.

If you haven't heard that one and God would trust us bring data. My personal favorite is a bad system will be the good person every time systems thinking right? And then for those of you Baseball fans out there, maybe not an angry fan the Yogi Bear of systems quotes.

Every system is perfectly designed to get the results that it does. There you go. So I told you I've had written this book 10 year later love.

Let's 14 months. I've been real serious. The other thing if you do decide to write a book I've written other coauthor books is the first book I've written all by myself.

That's a different experience. They say you write your first book for yourself your second book for your audience to this. You're right your first book yourself you write your second draft for your closest friends who almost become not friends because they trash your book and then you write the third draft for everybody else.

So just to give you the quickest damning biography that you've probably ever heard in the history of anybody who's talking about them. I'm just gonna go through this really quick because I really want to talk about like what would Deming do and in my book the last two or three chapters? Like what would he do if he was alive today?

If you looked at all the stuff that was going on. He probably wouldn't like us box. But hey the he was a mathematical physicist.

So that's what he got his PhD basically in 1920s. Right. And here's the interesting thing.

It was happening on what they call the second Scientific Revolution Newtonian to Quantum. He was being baptized at a completely, you know, if you heard like deterministic versus non-deterministic, like that's where he was learning so you can see that in a lot of his ideas and not deterministic. He was saying armchair epistemologist.

He actually was really interested in a uniquely American found philosophy called pragmatism. And again, I'm just going really fast. Like if the books out I go spend a lot more time in this and the other day.

His business card said he was a statistician. You know, he was asked one time. Like why does a mathematical physicist put statistician on his business card?

And he said well, I always been interested in the theory of errors. And I I was totally squares by the best. Again, this is a very short presentation.

But like you can sort of see where his head is going. It's probabilistic. It's not deterministic.

I use a professor. He was a psychologist. Again armchair, he studied psychology.

He watched he was a humanist. He really was motivated by the the joy of work people. He was assistant thinker here's an interesting thing.

The founder assistant thing is it's like terrible guy. I wouldn't mentioned he's a terrible guy. But like there are a lot of terrible guys that were patiently became great great science, and he learned General System Dynamics was the birth of systems thinking he was also in top secret project in Aberdeen during World War Two working with no weiner.

So again, I know I'm probably throwing a lot of stuff but like that was you know out of the sort of radar labs. agriculture probably most famous Is his work in Japan most people who do know about animals hear about he created the miracle in Japan. MacArthur send him over there after world two to help rebuild.

You just can't imagine. How terrible I mean like roads telephone water like he was sent over there to create basically industrialize. And then you know, he said devopsis because I mean 10 years ago a friend of mine, you know said John all the stuff you're talking about all goes back to Deming like that who Damn who and he said just go read demings 14 points.

And when you see that like if you're following anything about devops or SecOps, it's glaring. They reach out of the page. To jump at you any Futures, which is I'm going to sort of Express here.

The thing I say is cyber like, you know, I I'm doing I just moved to a college town and I've got I do stupid things right? I'm trying to start another business with some guys. I'm doing all this stuff and decided to to teach a class on Cyber to Old retire professors, like what am I doing?

But but I started rocking the history of cyber and it's just like all over the place. So guess what? I'm not gonna wait there right now, but for me cyber's future I'm sticking with that one.

Okay, and like there's a bit if you want to go Google at 40 times and see a different variations of it. So you probably have heard of this like Mark injury said in 2001 marketers and found our Netscape, you know, probably one of the most prolific investors today and decent Horowitz said that suffers eating wealth right like he was spot on No Doubt It's clearly I think it was 2015. But here together touch Corman somebody you should follow he's pretty awesome white hat hacker like extraordinaire.

We did a presentation together and he made sort of the software is infecting the world. Right like I wouldn't disagree. And you know, I don't like he was he he was at sisa for a while and he he actually worked on the cold chain project for the covid actually, but but since I put out I just in the last I think since 2020 they've been 10 emergency directives.

I mean there was one three weeks ago the VMware everybody started locked for Jay and then solo Winds, of course. The inmates are running the Asylum like it's it's I mean the point I saw recently. Oh, you can't do that.

The point I was making earlier and as somebody said this I was at swamp up a cup two weeks ago and you know, it used to be like security for people who weren't in security. It was like everybody else's problem. Right, but now like when you have this kind of stuff popping up deadly like who slept the night that the zero day for log 4J who slept that anybody like slept really?

Yeah like and then like solarwinds hit us like in a creeping a slow walk. But but the point is like this is in our face now right and like, you know, what's one thing to think about security as something else the security the word things but it's like coming from and it's not respect to VMware because it's anybody like it's kind of from VMware. I mean, you know, there was you know, like like it's like the things that we trust.

Are now being sort of part of our job and have to face and deal with you know, I think it is a little bit of a change. But here's the thing, right? hold everything else security is now gone to another level.

It's messing with Maslow. Right. It's messing with our health food water and shelter.

This is a whole new ball game. This isn't that people can't use funds for data, which is a terrible thing. This is like we can't get water or the water might be poisoned.

Right and you know, you know and you know, let's talk about like industry standard like we need to figure out how to get all the other people that don't come to this event to figure out how to do the things that we're trying to do because it's killing us literally just quickly just for shock and all right that's pacemaker. And about a million a year. There's a significant amount that still have a three-coated hardcoded password in them.

You know the stories of insulin pumps are legendary. You know just somebody, you know Bluetooth in but the the Hospital infusion pumps. So they Josh Corman and his team basically put together a hackathon in a hospital patients and doctors and they were able to I mean these things are all like admin password no password.

They were able to dump like they're not a real patient but 30 minutes of medication at 30 seconds. And then like let's just get into the cloud. It'll be fine, right they have these oncology systems now that are cloud-based.

So there was one this year and a couple one that went down and there was no Radiology. Right. So now chemo patients had to be rescheduled.

Right because I mean that's a pretty tight gig, you know, you're you're chemo scheduled and now all of a sudden the whole week has to be reshuffled, right? So Um, there's some research there's 5600 hospitals have no security staff. 56 hospitals in us have a security person let alone a VP or a director or seesaw.

so I don't know like This Coast thing but like from October last year to like January you couldn't get cream cheese. the East Coast I would say it was. Just I love no relax, but not tragic in the grand scheme of things, right, but it was a ransomware.

There was a second largest supplier of basically ingredients for cream cheese. And literally on East Coast you literally couldn't get cream cheese. Now the baby formula is not a ransomware, but just again.

You know step over two steps to the right and it's not cream cheese and it's something that basically is life sustaining. You probably all heard meat packing, you know, it's sort of a a revolution ransomware, right? It's got to a new level ransomware as a service.

But here's one you probably heard of drinking water in Tampa Bay Tampa, but would you probably didn't know is it was like three days before the Super Bowl and it was less than 10 miles from Super Bowl. So somebody was literally trying to poison and it was an amateur. Because they were used like a remote wear and the first time the operating Authority just thought was something weird second time.

He watched them like upping the sulfur or cyanide that basically you know, like oh that's not cool, you know Colonial reach. Here's the one that you might not heard of the first litigated death to ransomware happened in Alabama this year. So a woman went in for birth.

And it turned out to be complicated Earth but radio oncology was down so you can radar so they had no idea that was going to be countless. They didn't have the staff. To newspapers where they otherwise would it was a ransomware they didn't tell her.

You know, what is this lady need to know about you know ransomware, you know, well, they didn't x-rays the baby died about a month after birth. And now it's the first sort of litigated or death death for real death. So to think about Deming and like what would Deming do right like The the core of deming's ideas based on this idea of what he called profound knowledge.

And this is important. Because you know we can talk. With Reagan on Esperance we can talk about those seconds we can talk about response.

We can talk about a lot of different things. but the end of the day the way we saw real problems is not just that I think that Deming pull this from the sort of the the world to figure out to tell you about it was like he invented this But basically what he said is to solve any complex problem and by the way everything we do. Is a complex problem we live in a world of complexity.

Right that you have to look at it from four lenses. First is theory of knowledge. I talk about epistemology, but it's much simple in it.

How do we know what we know? Right, and that's scientific method. We go back to Francis Bacon.

Right. Yeah, Demi called the plan do study act, you know, really goodness industry plan do plan do plan do plan do. This follow-through of basically, you know and its roots is epistemology.

His favorite flavor was pragmatism took a how do we know what we know? Two, how do we measure what we know? Now this is comes out of industrialization, but it could be I mean basically it's called statistical Bots control and it's about variation and the quality of manufacturing but it works really well in software particularly nuclear power plants, but but it could be any form of measurement and more importantly non-deterministic measurement things like Lee Scott like statistics like it's not yes or no.

It's probabilities third and this is what I think is genius came in which is Like I can basically analyze the complex problem with like everything. I know about scientific method and thinking and continuous learning like plan due did it work change the experiment do it again. Let's continuous Improvement.

Right? I can measure either observability. I can do statistical process control or any variant of that.

But what I can't do is I can't change your cognitive biases. If you don't believe there's a part of your organization doesn't believe something me just throwing all these facts at you is not going to work. And more importantly there are people that are intrinsically motivated and there are people that are extrinsically motivated.

And if you don't understand that. No matter what you do, no matter what chart no matter what data you show them. You know the way I rock this the first time was it was a study on hand watching in hospitals.

It is simple as it sounds and as silly as it sounds the conclusion was that if people didn't believe that feeling washing hands kills germs. Then it didn't matter. What data you showed them.

And last but definitely not least is to encapsulate all this in systems thinking. System thinking is the core. Of like of what we need to understand in today's world, you know, it's just you know, there's plenty of books.

Somebody asked me for my reading list the I'll go back to the old original Peter sangay if discipline, you know, like, you know, there are books. Danella Meadows another great system thinking writer, so Again, this is a very short version of the four. But like that's how you look you.

It's like the way you take a picture, you know aperture and all the things you have to sort of align all these things and if you're not you're going to be sort of imbalance. It's like a Chinese medicine so. What I did in the book is how many people heard the night capital?

Tobacco. Oh nobody was good. So night capital is interesting right in 2012.

They're the largest Trader stocks on NYC and NASDAQ. They were electronic funds so they did basically Black Box trading they traded on on the exchange. There was a new system that came out.

3 billion trades each day an artist for 2012. They lost 440 million and 45 minutes. They were basically had a business in 24 hours.

Here's the back story. They were they were installing the software on eight node cluster. And for whatever reason, you know, 2012 will give him benefit out.

Like I'm still old Chef fan used to work there. Like they were using Chef pop at her ansible or whatever. Hey, I don't know and you know my good friend John.

Also, I will yell at me right now say these are all kind of factuals and they are I wasn't there and I didn't know this is I'm using it for training purposes. So they didn't install the new software on the eight server. Great, so there was some test software.

on all eight servers and it was offered many years because there was a flag that it looked for common Fleck. They repurposed this flag. And so let me ask this question in general how many people trade stocks buy and sell stock anybody?

Do you think it's a really good idea to buy high sell low? The terrible idea right? Well, that's what this testing algorithm did.

It was a stress test. It got turned on in production. And it literally they lost 440 million.

In fact, the I don't know if you heard flash crash, they actually put circuit breakers in after the flashcard and that's because of a price of a stock drops like within 10 dollars or you know, 10 minutes or something like that. They didn't have a circuit breaker for volume. The volume just because it is one sort and it got worse.

Just what did again I'm being kind of factualist, but just for fun the Geniuses do they roll back the other seven service, but they've got to turn the flag off. So now you're at eight servers in basically latency Arbitrage. Like making production trades at by high solo.

So what would Deming do if he had this Debbie died in 93 by the way, so so if you look for lenses, he would say in fact the SEC sent to cease and desist now, for those of you I heard a couple of people have software companies and stuff like this just approach it. You never want to get it see some distance from the SEC. Like if you're gonna start a business like not you just don't want that to happen the so in the theory of knowledge, you would say that how do we know know?

The SEC said specifically that they had no finding that there was a second technician to review the code in installation. So there's no there was no sort of pairing on a pull request or s*** two minutes. I'm gonna steal three minutes Mark.

Sorry variation procedures the They were not like they didn't have they didn't have that sort of check. They weren't doing sort of variation. I talked to a friend of mine who's in another high frequency trading.

He says yeah that can happen. But we recorded in five minutes this we got we have data. We know what's abnormal psychology.

This is pretty interesting. This is saying you happen to Boeing 37 37 37 Max right like the SEC claim that no, but that were people that knew this was a bad idea because they were rushed it and nobody felt empowered psychological safety to say something and then they're sort of their adequate written. They just didn't have a systems approach.

So just quickly the demings 14 points. I only cover a couple I tell you about the Schreiber Foods. This is an interesting one.

They don't even have they didn't have they had a director of security was from Hardware. I checked periodically, they still don't even have a VP or CSO. This is a ransomware like yeah, so yeah after this after that breach, thank you for making that face, right?

Yeah, it's horrible Shadow Elites like she's not here. Unfortunately. She's brilliant.

Google her read everything she says she's incredible. She does this thing called adversary analysis. And it's just so much, you know instead it just sort of like try to get all the vulnerabilities out and do this and do that.

How about like doing proactive stuff trying to analyze who the guys is? All right. I'm going to P1 p2s.

I got to do this one Mark I got okay. 9. No silos, right?

So the extract speech if you read the congratulate report on it, it's really interesting to see so reported the chief legal officer. So when you see so was asked did you think it was odd when they hired you to report Chief larger legal artists? She said no.

I just figured they knew what they were doing. So then she gets asked later is why didn't you contact the CIO when you found out there was Pi breach and she said I just didn't think about it like you know, so that's Conway's law that sort of silos so quickly Deming would hate. Okay ours.

I hate him. Absolutely. And in fact, I think you would dislike as polish but for a little bit different he hated the idea.

So what he said is there's a quote from from his first book the car transformation west of style of management requires that manage the leaders. How about that focus on outcome management by numbers standards specifically desert defects praisible for us must be abolished. The bottom line is whether it's a kpi an MBO an MB and b r It's people are going to game it.

They're either going to fail and be disappointed or they're going to game it or they're going to do it because they threw pixie dust in the air and it happened to work. It's not the way to lead. What Deming said is basically by method?

Like what's the method? I don't care what the results are, you know, the famous quote right give a person a fish and feed him for today teach a person to fish. So he was about not he was in about the results or the objective.

It was about the I teach you the method. So that this could be sustainable. Do you know how you got the results?

Can you explain the results? Can you explain the results to other people? Let's face it like okay ours or just another pattern.

Of you know, like they're me my team my division. Like it's about us and our bonuses and we we get a pat on ahead. All right quickly, he would love SRS.

Because history completely divorces. It's not like I'm selling Indian history, but he would love SRE because it isn't about the individual. It's about the service.

It's totally decoupled. It's it's a systems approach. It takes in multiple indicators it has method anyway, so And he definitely wouldn't like kaylox.

I think that's all I got to say to you. Like Kay locks are just a great example of literally motivating people without the method or the means. anyway That's it.