The Void – Courtney Nash, Verica

This is Textron TV. Hey everyone, welcome back to techstrong TV. Our next guest on texture on TV hasn't been on with us in about a year.

Now. It's Courtney Nash Courtney is with verica and well, she's with verica now, but Courtney has a long distinguished career as a real honest to God. Researcher analyst someone who really kind of understands and knows how to put together analysis researching, you know gather various data points to to form.

Opinions and you know show us the way Courtney. It's a great. It's great to have you back on and welcome and thank you for coming on.

Thanks so much Alan. It's a treat to be back. Yeah, it has it's been it's been quite the year quite quite the year for our industry as well.

So yeah, we're back with the the second annual void report. So I'm excited to talk about that very good incident database. So Erica open incident database void before we jump into this year's report.

I I like I feel we need to set a little bit of a Foundation, so I mentioned, you know, you have a pretty distinguished career if you want to. You know, if I don't embarrass you if you wanted, you know, give us the details a bit on that and then maybe a little bit about verica and why you know last year was the first year of the Void, you know, and and what's that all about? Yeah.

Absolutely. Yeah. I mean I've been in industry for for quite a while now.

I started working at Amazon when it was mostly just a Books and Music store. So that was quite a long time ago long before AWS was a glimmer in anyone's eye. I have to say I spent a lot of time at O'Riley media, which is where I really honed a lot of my sort of trend research, you know, figure trying to understand what's happening in industry, and that's that's actually where I got interested in.

How systems fail and and and where things break down and and some of our I have a background actually in Psychology and Neuroscience. And so I spent a lot of time thinking about like how we think about computers as humans and and where some of our Some of those assumptions might be might be as you know, go astray or might not be founded in actual reality. So when I went to America, I started really being able to research pretty pretty complex systems and that led to the creation of the Void which is this the varica open and incident database I Stumble over that all the time we've collected together now about 10,000 public incident reports.

So these are things that they could be anything they could be something the media rights about but a lot of many of them are status page reports or full-blown sort of post incident review post-mortem types of things and our goal was to collect all of these in one place for for a number of reasons, but the biggest reason because nobody had done it yet. There's lots, you know, there's lots of security breach databases out of there. There's lots of other ways to look at data, but this never really existed before and and I've always been inspired by how and completely different industry did this so back in the 90s our aviation industry at least in the United States was in sort of a state of Crisis that a terrible safety rating people were actually nervous about flying and the pilots actually started talking to each other about Incidents and so it was this sort of practitioner-led, you know ground up kind of thing.

It wasn't regulation coming in and saying, oh you've got to do this, right they all look around each other. We're like, oh boy, if we don't figure this out, you know this could this could just get worse. and I believe that the same kind of effort in our industry can help us because everything relies on the internet now, right because safety critical things for lie on the internet now communication just all of our Lives rely on the internet and we want it to be safe.

We want it to be reliable and anybody who works sort of at the sharp end of that nose that it's not as safe or reliable as we'd like it to be so having all of these data in one place allows us to look at them more systematically allows other people to look at them. And so that's where the report came from last year. We started looking into some common.

Assumptions some sort of what people might say gold standard metrics and and found some of them lacking or like the data didn't back up some of our a lot of our beliefs and so things like the duration of these kinds of incidents. The severity of these kinds of incidents. They don't hold up to really solid analysis.

So my favorite one right now is mttr, you know meantime to respond right or remediate the fact that we don't even all agree on what are is might be the first problem. That's a clue. It's a clue that's kind of a gold standard in the industry and I feel like it we picked it.

Probably for good reasons at the time, but we didn't pick it because we were looking at the underlying data we picked it because we assumed that mttr would tell us about you know, the the reliability of our systems or how well we're doing at, you know it responding to incidents. And if you look at the underlying data, it's just not feasible. You can't take averages of these kinds of incident duration data.

They're so all over the place that the mean is meaningless. So we really confidently demonstrated that this year because we have 10,000 incidents instead of you know, a few thousand that we had last year. And so I think that's a big one because I mean people kind of try to fight me on this one Alan when I you got it.

I mean, look I I'll be contrary and I I've been around the industry a long time, right? I I think the whole meantime to I always called it remediation, but I have heard people say response. But the meantime to respond to remediation I think was a knee jerk reaction to many many reports that were coming back out 10 years ago and longer that said Oh, you know this this was in your system for 18 months before we found it.

Well, this breach happened 12 months ago eight months ago. And and now you first ringing the fire alarm. It's a little bit late.

I mean you're closing the Barn Door. yeah, after the cat, you know the cows ran out and unfortunately, that was the That was the norm. We did.

We didn't find out about a lot of these incidents. Until the the proverbial, you know, what hit the fan. Right.

The the information was already used to create fake idea accounts or the money was transferred to 15 different bank accounts, or you know, what was security things it's even I think it's even more farfetched to use that as a as a number, you know, like at least in the case of oh our site fell over you don't know when it started necessarily but you have a better sense right there and then something's happening. But but to say that you're getting better or worse at it is it's the data don't allow you to really do that. And and I argue it doesn't matter, right?

It doesn't matter what your mttr is because the only way you learn about what's happening in your systems is by going and looking into those incidents right talking to the people at the sharp end of the wedge, right who run those systems and they'll tell you. Oh, yeah. Well, oh yeah six months ago we do priorit.

Is the work on this thing because we had to go over here. We only had four people and this thing was sitting over here just like we knew it was falling over but now here we are or like there's a million things that could contribute to these and that one number just doesn't really tell you very much about what's actually you know, sort of happening in your systems. And so we confirmed that from we looked at that last year.

I feel like we confirmed that this year but the new one this year was to look at severity and direction right? So I'll ask you this like would you have a would you have a hunch? Let's say that that longer incidents are somehow worse.

For better like what would you know, what do you mean by longer incidents? So let's say you go look at all of your data and you're like, oh well, that one went on like, oh this one we solved it in, you know, oh we figured it out 20 minutes, you know rolled it back and it was great, but then you had one that went on for three days. I mean, so You know count your intuitively I I don't know if that's indicative of how serious it is.

But certainly I mean on its surface you say. Oh my God, this is existed for so long. My mmtr is gonna go through the roof and and you know, and it's really bad.

Yeah. There's no there's really no relationship between those. All right.

So having unfortunately live through that. Yeah. Yeah, I I don't because I think what happens is a lot of times you have an incident.

and it's just the first, you know, it's that reconnaissance if you will where they break in or whatever and they they're kind of You know checking out. I mean the whole thing of how you know if you did the The life story of a data breach are a security incident. You know, it's it's they're usually not hit and runs hit and runs are kids, right they go in they swipe something in there out.

Yes, usually a much more fiendish plot right where they're gonna go in they infiltrate. They they set up shop. It's like termites getting in your house.

You know what I mean the first day they're there no big deal, but six six months a year later. The whole place is falling down. And it's so to me that's kind of what it is.

Yeah, and I mean with the with these like so with the void are more what you might consider like availability incidents. It's supposed to experience right? So you get it.

Yeah, but but even that it's like we were able to show statistically that there's no relationship between what the kind, you know, so you let's take a status page report for incidents. Right and it's a lot they tell you A lot of times it lasted this long and we called it as of whatever like a Sev three, right? Which means like oh not so bad.

You know, it's weird. I think it's like Defcon levels. so like a four three is you know, and the one is like this one is like right red light.

Yeah, so so we were able to correlate try to correlate those kind of severity ratings of incidents with the duration and we didn't find any So you can have short ones that are bad short ones that aren't bad long ones that are bad long ones that aren't bad. Right? And so that's another one of those sort of indicators that like, we think severity might give some meaning.

I mean the people have these dashboards in their internally right where it's like, what's your mttr? How many seven ones have you had this month? And it's like again, we're chasing these sort of meaningless metrics instead of looking for the meaning in our systems right where where we actually can find out what the spooky stuff is.

What are what are you know sort of wrong unfounded assumptions are you know gaps in knowledge and all those things those numbers aren't going to tell you those things. And and so that's really where we're pushing with the void is is to get people to focus on the people in the system. First of all, because we've collect we we collect these technical metrics right meantime to resolve severity duration.

They're they're so tempting to us as technology. It's quite right the numbers, you know, yeah makes us comfortable. Yeah, you metrics.

Yeah, exactly, but Courtney work, you know, I said you're CEO of texture, right? You know what my like not biggest fear, but unfortunately the fact is that we find out about a lot of these incidents from customers and I freaking hate that. Hate it.

I hate it when like someone I know writes me says Hey shimmy, you know, if you go to this URL things don't look so good. You're something's wrong. And it happens right?

Because look we're running. I think we were responsible for 40 different websites these days here, you know that a part of the text strong think and and you look in it. So men, I mean so many different aspects to it the search the default search isn't working right.

So when you type in Courtney Nash's name, I don't get anything about Courtney naturally. She never existed on our side even though we've done you know, Little things like that just pissed me. Off tell me the systems are complex, right?

No, I know I know and and we don't. I'm sure bigger companies have better instrumentation. They still have problems.

That's everything we talk about in the report is everybody has incidents. No one is immune to this unless you have like a single page website run by you and even then even then it's just right. Yep.

And so but we forget about the people in the systems. We focus so much on the technical aspects, right like oh, you know this that the other kubernetes stuff and hooked into this that like we should we need to but it's the people that build those systems that that have the sort of latent knowledge of what how things work and how they don't work and we aren't collecting metrics on that. We're not looking at I look the big news of my world the less week or two is this wasn't cool Chad APG.

Okay. Yeah. I've got Riders.

Freaking out writing, you know hate messages about why this will never replace that and we've all been through this. I don't want to be replaced by your computer and you know, the whole premise of devops is we try to automate more and more and more because that's the only way to keep up the data sets are so big the speed of business is so fast. Now you need to order me.

but who writes the Automation and I think the funniest thing is people really believe this is another myth that I don't have any data on but people believe that automation, you know, like someone has to write the Automation and then it doesn't just keep working. And the reason it doesn't work is because you write the automation on a system as it exists at that moment in time. And from that moment forward that system will never be exactly like that again because of those pressures right?

We're constantly adding pressures to these systems and and we're so most of the people who build and maintain them are so good at that then the business go. Well sweet like more pressure right? Like yeah, the system gets the more pressure we put on it and then it starts breaking in new and funny and weird ways.

And so that there's all kinds of, you know, great work around Automation and the ironies of Automation and how our beliefs there are also founded in. False hope that's just it's not full talk here. It's not false.

Hope it's good hope but it might be false. Hope you know a new hope. I'm reminded of Star Wars but here's here in my mind.

Here's the fundamental and this is what separates. There's two kinds of people in the world Courtney, right? Oh, we're getting philosophical now.

I'm gonna go all philosophical on you. There's people who believe that by doing all this Automation and getting better at our technology and doing all these things. We're making it simpler.

Right for the humans. And there's another kind of person who says the more we do here. It's getting more and more complex and almost beyond the grasp of humans.

Right and it goes to is the universe going to expand into all the stars go out or is it going to contract back again to a big bang? Right? It's the same see this interview going here.

Look you never know where these are gonna go but But you know, so are we getting more complex? And that's that complexity mean and can somehow can we translate that camp complexity the Simplicity for people to wrap their heads around this? Yeah.

My my favorite analogy and Adrian cockcroft wrote the forward for the void report and Adrian used to was at AWS for a long time. I know agent Adrian SE. En some common sense, you know and and he points out that the the airlines that have the most incidents not accidents but incidents have the best safety records, right?

So the people who are the closest to those boundaries know where those safety boundaries are right and and they are very aware of how complex so systems are and that they can't control all of it and that there will always be unforeseen and explained outcomes. And and those are those that so I I fall in the it's getting more complex. You can't understand it.

You need to figure out how to Grapple with it. Camp and that's the kind of the kind of thing that we're trying to do with the void. And and I just keep coming back to like the people are are the Unseen people are they're like think about this.

How often does your system not go down. And why it surprises the academy rightly how you because those people are constantly going. Whoa.

Did you hey Jane. Did you see that over there that and then Jane and Amy go over and look at it and they're like, oh That's weird and then they you know, and then they fix it before it gets out and every day thing, you know, we use slack like most organizations and we have a channel in slack called engineering alerts. Yeah, you don't have to be an engineer to you know, the whole company has access to that channel.

And yeah, no, it's in every day thing Courtney. Someone puts this I'm seeing something funky over here. This doesn't seem to be displaying right over there.

And it's a it's a constant, you know, and again, we're not the biggest company in the world, but it's a constant thing. That's my point though. It did like you're no one is immune and and we should all be talking about this.

We should be sharing this information and that that's what the void is for. Right? Everybody should be writing up their incidents and and putting them in there and learning from them and sharing because otherwise we're all off in our little worlds right with our commoditized knowledge and that and a lot of us are having similar things.

They might be different flavors of similar things, but if we don't share this information if we don't get it out there and like openly then we're gonna continue to have bigger problems and to be quite honest, you know, we're already seeing a lot of talk about what regulation might look like more and more in this space Not Just security breaches, but just like You know downtime kinds of things. So I think I just read about something it was funny because you know in devops we have the Dora. Was Nicole Dr.

Nicole Ford's Grand and Jess humble and Jean they got sold to Google but they still do the door reports. Well, there's a new set of regulations coming out of EU called the Dora. Dora and and I don't remember what the heck it stands for now, but if you go Google the Dora from email and it's gonna it's gonna put in some, you know exactly what you're talking about.

It's here. Yeah, and and this is gonna have an impact on on all of us. I don't know what it's gonna be between I thought the folks at door and maybe had a trademark, but we'll find out in any event.

You know, we've had a great conversation and I made a very personal to our situation and where it is, but for people out here who want to get the void report this second year of the report. How did they do that Courtney? community because we can have super weird URLs now and there's a big button that says get the report and it's super easy.

We email it out anybody who wants it and we also have like we have a newsletter that's going out. Obviously we have blog type stuff. We're launching a new membership.

Program and an advisory network in in 2023 so different ways for people to get more involved in this not just, you know, reading the report but sharing their incidents or bringing that kind of approach and methodology into their own organizations. So there's a lot more that's gonna be happening next year on that front too. But yeah, hit the button and you get the new report all the details around.

Mttr duration severity, we look at root cause analysis and and the different methodologies that people are using and some some that we think are a better. Mental model for how to you know approach these kinds of incidents. So there's a whole bunch of additional information in there beyond what we even talked about today.

I love it. Give me the URL again for maybe those who aren't didn't catch it. Yeah, it's void dot Community boy dog Community.

Yeah, what about people who want to contribute? I mean, I know you have membership coming but look until all that set up. I I'd like to contribute our incidents to the to the list.

Yeah, definitely do that. I double checked I should know this. It's embarrassing.

It's the void that Community you guys didn't you know fix that for me. That's right that rocking chair Community. There is I just go there all the time.

There's also a form to submit one by one kinds of incidents if people have a whole lot of write-ups. They can there's a contact form they can get in touch. It's been just me.

I Ivory Tower for the last year so so I don't have like a fancy API built yet a more. We have a bunch of those kinds of things in the works for next year as well. But I will gladly work with anyone who wants to get their incidents into the void and make sure that that happens that would be great.

It's worthwhile. You know, I mean, this is a I'd look I love the idea. When we spoke to you about this a year ago.

I still love the idea. Keep up the great work Courtney. I'd love to hear more and you know what now that covid's receding.

Hopefully, maybe we'll see you somewhere. But well, it's not receding. We all went down you proceeding but someday We will do this in person and sounds good to me, Courtney Nash.

the void dot community go get this year's report. It's a great report. io, right?

Yeah America that I owe. Yep. Excellent.

Thanks. All right, this Alan shovel. We're gonna take a break here on Tech strong.

We'll be back in a moment.