Incident Management – Courtney Nash, Verica

This is texturing TV. Hi everyone. Welcome back to Tech start TV.

My guests for this next segment is my friend Courtney Nash. Of course Courtney runs the void. She'll tell you a little bit about it, and we're going to talk about You know what's been going on some of the things been going on lately out now World Courtney's always a great person to talk about it Courtney.

Welcome. How are you? Hi, Alan.

I'm well, thank you little jet lagged and I was traveling a bit, but I'm back and Ready to talk about all the fun that we've been seeing. So well, we're going to talk about I think traveling as well. But yes, I mean the good news is we're all starting to travel more excited and Seeing things after the last three years though, you quickly if you're experiences anything like mine, you quickly find out the airlines haven't gotten a lot better in the last three years.

They pretty much the same thing. It was before covid crowded points seats. I like to give the airlines a lot of credit because I feel like we've learned a lot from as an industry, but Having a rough go.

I'll say right now. Yeah be fair. It's not just the airlines.

It's no everything around them. But let's not before we get to that Courtney. I got to come back and I said Courtney Nash from the void people who aren't familiar.

Why don't you get tell them a little bit about the void? Yep. So the void is the vertica open incident database.

We're about two years old. Now, we collect incident reports for software. So when things like you might know about right now the FAA Southwest Airlines Ticketmaster all things we might talk about when these Services fail.

We if anybody writes about it, we collect all of those into this this database so we have about 10,000 reports right now. And our goal is really to just raise awareness of software related failures because as you were sort of alluding to like our world relies on software now everything our planes our concert tickets our banks like an all relies on software and we don't really have a collective understanding of Why these things don't always work and they won't always work I think is yes is one interesting part of it. So so we've been collecting these reports.

We've just put out our own report the void report 2022 the second one where we look at data that we collect from from these reports and try to sort of push our thinking forward in the technology industry about how we think about Software failures why these things fail? There's a lot of in my opinion sort of misconceptions or myths and and common ideas that we might have that are not necessarily. I mean they're grounded in what we feel like is reality, but I think we have a lot to learn about how we think about failure.

First of all just as humans, but in software systems, it's really easy to blame people for things and you know, so for example, the FAA situation has ultimately sort of come down to one engineer who, you know pushed out the wrong database file and then all of FAA goes down and we don't know much more than that at this point, right but what you know, what I thought was interesting about the FAA situation was they didn't blame that person really they didn't say, oh, you know, they sort of said our systems ultimately should have handled this better. There's a lot of should have would have could have but I thought that was really interesting because it's very common, especially when the issue Come down to a person which happens a lot that we used to just blame people all the time for these things. And so that was it's a complicated situation the FAA thing obviously, but I was really sort of pleased to at least see that they said this wasn't just one person's fault.

This was a system. Now as you know, lots of people have criticisms of that system should have had should have been updated sooner like a lot of that but the void looks at that we look at all kinds of implications of software failures and and how we can learn from those and collectively and as an industry people who work in software systems get better at it. And that's where I say.

I love what the airlines used to do. The airlines started sharing their incidents a while ago. This is back in the 90s and they got better right?

I think they got better. So maybe there's maybe there's more to do now there's work to be done for the airline industry, but I still think we're lagging even behind that as a software industry so that our research is really focused on looking deeply into these kinds of incidents versus at the surface facts as much as we can. So that's the newest report talks a lot about some very technical details of software incidents like how long it takes for things to resolve and we just I think we talked about this last time so like yeah meantime to resolve as is considered a very gold standard metric red herring.

Well, I mean, here's the thing it feels right right like it feels like okay. Well, how long does it take us on average to fix our software systems? Like I don't want to I don't actually think they're that it was foolish per se to choose that as a metric but we didn't look very closely at the underlying data.

And now now that we have a bunch of these together we're able to do that right? We're able to actually look at the data and underneath right? How long does it take for each company to resolve all their incidents and it's it's not conducive to something like mean time to resolve.

And so that that's one of the findings that we confirmed in in the 2002 report. We really would love our industry to move past using that as as a metric because it's kind of just lying to you. All right, it's red herring.

I mean, let me let me weigh in if you don't mind Court I would love for you is so think when it comes to incidents, especially around incidents with computers internet technology. There's this like, you know how hard it is to and we try the economists try to measure consumer confidence. I think there's a consumer confidence type of thing with computers.

Right? And this FAA thing. It was a perfect example, right?

We were talking off camera when it first when I first saw the news around this FAA thing, my first thought was oh my God. I hope this was it a terrorist some space state sponsored packing, you know, a real cyber incident if you will, right because that was the response from all the way up to the president saying yeah don't believe this is you know, yeah, it's well that that's your first thought because It's why we can't have nice things in general right where we're still little. We're we're just conditions elections.

We're pretty paranoid for good reason exactly. and it's almost like so when the news came out it was A bad file in a database I think for the most part non-tech people. Shrugged and says look that's computers for you.

You know, I mean that that's kind of the common thing is yeah, it's computers man hit Control won't delete and start over at something and we've almost become. I don't know if it's desensitized or conditioned to expect these kinds of things when the fact is I'm going to play the take the current contrarian role. When you look at the amount of planes that are in the air every single day over the US the fact that we don't have more serious incidents is a testament to how well these systems run that's not okay.

So in my opinion that's not a country, but well, you know, and this is the thing about about humans and perception and outages and these kinds of things it's very easy for us to latch on to the failures and the bad things and and those get very publicized right because you know, the the good news the best news articles aren't like US Air industry mostly working all the time like that girl. So so it's we it's a perception problem. 9% Like that's right.

No our brains work. And so we have to constantly be kind of fighting against against those kinds of instincts and we talked about that in the report too. Like it's not easy to do but it would be great if we could study.

Maybe not all the successes, right because nobody has that much time. But but study the near misses, right and this is also an airline industry learning and other Industries too though, and you know sort of Maritime Industries nuclear power like where did you almost? You know, can I swear on this program?

No, sure. Yeah, it's weird don't. Where were the oh s*** moments.

Right, right because those are so tallying like it didn't quite. You caught it. And and what was that what was happening in that moment where you caught it?

Because that's your system largely working as designed right the things we do to make it work the majority of the time only ever once in a while turned out to be the things that make it not work and they're the same. System we don't you know, so that's well. So two thoughts on that first of all, there's a very thin line between working and not working in a lot of our systems.

Okay operate at the boundaries. Yes. That's where the money is.

That's where the opportunity is Right. Mmm, and we get very good very at performing in those. Little nominal spaces in the Great Space.

Yeah, I know in the the other thing is though that you know, we talk about critical infrastructure. We talk about infrastructure infrastructure upgrades. You know.

You needed digital infrastructure upgrades. As much as we need roads and bridges many times. Yeah, I mean, I think they're the metaphor works.

I think sometimes we still metaphors and technology and we've used them or misuse them but it is it's it's critical infrastructure. I think the interesting thing for example about the FAA case or the Southwest Airlines case what happened sort of right before the holidays and was had a similar feel to it is well, let's set the FAA because that has all the government do they get enough funding, you know angle to it. But one of the biggest criticisms about Southwest Airlines is not like we did oh your technology so old but you have to go back and ask why why is it so old and why did Southwest not invest in that over time.

There's gonna be a million case studies. I think on this Southwest Airlines situation because it wasn't it was a business failure. It wasn't a technology failure.

And if you ask anyone who worked in that, you know within that company or within that industry same thing Southwest Airlines that operated at a very very narrow margin compared to even the you know, so if you The slice that the airline industry Works in of performance near the margin where they're going to fail Southwest Airlines took that right to the edge and that was that was part of their business model. And and yeah, but they were always known I mean Courtney Southwest were the smart cookies in the airline industry, right? They like to fuel prices for three years.

They innovated they out innovated the big guys and you know, they would kind of the Avis. We try harders of the airline industry, but now we came to find out well, yeah, maybe they did all those things but they didn't really update their ID as much as they should have could have their idea first just as well as it they wanted it to until it didn't yeah and that they know business level right now at the at the operational level people have been climbing for them to update it because they were the the people in that system. We talk a lot about people and they were the ocean people very people are providing the adaptation that the technology won't and so as far as the business is concerned still working.

Until the entire pressure on that system out weighed what even the people could adapt to and then you see the whole thing sort of fall apart. So so what yeah back to your near Miss. Oh s*** stuff, right?

I'm sure there were near Miss. Oh s*** stuff going on for years that they say. Hey, we Dodge that bullet Dodge that one too, and we're okay.

We'll get through this right and and then but then the crows come home to roost and and I mean if you talk to Pilots or ground crew or other folks at Southwest, you know, and even a union had been screaming at Southwest for years to update but they you know, we we as humans and then companies as entities tend not to react until it's really bad, which is unfortunately, so let's let's come this guys. Any other recent incidents or stuff that you want to? Well, yeah one that we put in the void last year late last year was Ticketmaster.

Did you know so I'm gonna tell you something not for nothing, but it was it was through Capital One not through Amex and I happen to have both but I put in my my requests through cap one early on for the pre-order and I got a very nice note that basically said you lost we didn't choose you and I immediately thought well, they must be age discriminating, but I did it. Hi between you and me. I wasn't losing sleepover.

Not seeing Taylor Swift anyway, but but what it was was the my niece asked me to get them truth be told for her. It's usually what it is. Yeah, and so I I said, okay, I didn't make the presell.

Let me see how this works. And then I saw what you know keeping in our Motif today. What a s*** show it was and I was like this is this crazy they You know the absolute power corrupts.

Absolutely. They need a good competitor to keep these people straight because this don't work. So that's the argument although and you know, they kind of the funny thing is Spotify was like whoa, because the Ticketmaster thing took all the attention off Spotify who always fell over momentarily and they were very prepared.

They knew it was coming but even still there's there are forces that are hard even when you're well prepared, but the ticket situation is interesting. Have you been following any of the Coverage the Congressional coverage. Yes.

I certainly watching. I don't know, you know, so there's the whole monopolistic side of it, which I think is yeah. I'm not gonna necessarily disagree with and I certainly hope but what's been fascinating to me the details and the discussion around Bots.

Um and Congressional and well, first of all first and foremost Taylor Swift has United the country. Yeah, no doubt only thing that Democrats and Republicans can agree on is that everybody wants me we want our tailor tickets I get you. So that's what it comes to and and our country but I the bot thing is really fascinating to me because it's so we've sort of the Congress is kind of latched onto that and why haven't you developed algorithms for these thoughts, you know, and it's I'm not saying it's a tough problem.

So Bots are a really big problem depending on who you ask 40% of internet. Traffic is Bots and the interesting thing is Bots aren't necessarily bad. They can be good.

They can't serious a thought most of the most of the things you chat with for like customer service Amazon, where's my stuff? That's a bot. Absolutely.

So it's all contextual and relative right? But not even to mention this new chatbot, right GPT. I'm going there you can get someone else on to talk about no, that's a whole different show.

I'm not doing different show but you know, it's there's this sort of after the facts judgment like why haven't you done this and I think it's interesting because of course, they haven't done anything about the Bots because because of the the monopolistic right? So the incentive when you go back up to the incentives that tells you a lot about the technology systems and so it's really easy again to focus on the technology right to focus on why do Ticketmaster go down the Trap right? I'm going to stop you a second.

Yeah. It's really easy when you're some congressman who God knows what you did to get your your seat, but you don't know diddly about Bots and technology and you're gonna start telling the technology people what they should or would have done. Yeah, so it's fascinating to watch that play out because we regularly complain about how our representatives are not well versed in.

Oh, they're not you know. But what that's all that's a different show to Courtney. Sorry, but we'd like to do with the voic something that seems like a technological.

Thing incentives what what's actually happening? What kinds of decisions were made far away from the poor people trying to you know, make Ticketmaster run that day. And and so that's those are the kinds of questions.

We want people to start asking not just why did the database file, you know, cause it to come down because that's that's usually the easy story but not the whole story. So I'll be very curious to see what happens with Ticketmaster, but we'll watch I mean, right, you know, I knew this was a problem like last year because I'm so I have I have season tickets to football I get a lot of Yankee tickets for when I go to New York. I have like off down here.

We have Broadway Theater that you touring Broadway shows. I have tickets to that and then I have my regular Ticket Master account. So I have like five different.

Ticketmaster well five different accounts for tickets that I came to find out. We're all Ticketmaster under the covers and I had five different logons. So every time I would go to open You know my off my Broadway touring Broadway tickets.

com. Yeah, right and but they're different and I've said to myself this is this is this is gonna come back to bite someone suit because I you don't work don't make sense. Yeah, but it you like I always say out it's always incentives.

Like where are the incentives? Right? Well the incentives where they had a locked up.

Anyway, it wasn't sativized like to care about Bots. It was working for them. Right absolutely was and they never legislation solve that I don't think so, but I don't know competitive.

Anyway Courtney we're out of time, but it was so great catching up. We didn't tell people they want to get more about the void and stay up to date. Where do they go?

Yeah, so they go to the void dot Community, which is a very strange URL. I realize I'll give it to you. If you have show notes, you can put that in there.

You can search to their see what kinds of incidents are happening more recently, but I definitely recommend people go download the 2022 report. It's a it talks a lot about. The people in the systems these technological systems and I think it's really easy to forget about that that people build maintain and run these things and we have to pay more attention to them.

And and how they help make it work most of the time as we were talking about today and how they how to help solve it when things do go wrong. So hopefully if you download the report and we'll talk about some more soon. I'm sure because it's not going to go away.

Unfortunately Courtney Nash from the void the void dot Community. Check it out here on techstrong TV. We're going to take a break.

We'll be right back.