Exploring the Unintended Consequences of Automation in Software with The VOID’s Courtney Nash

Hey everyone. I hope you enjoyed Text on Gang today. It was another lively discussion with some strong opinions, mostly from me, I guess.

I'm sorry, I, I don't know why I get so worked up. I have some UNC channeled digression or something. Anyway, I hope you enjoyed, it's some good discussions and thanks to the gang members for coming on.

Um, the rest of our Tech Trunk TV show today is a great one. We're gonna start off with my friend Courtney Nash, who is the kind of main person behind the void and the Void report. You know, the Void at one point was part of er, it's been spun out at its own, um, it's own company now, it's own entity, the Void.

And their report titled Exploring the Unintended Consequences of Automation and Software is Out. Courtney gives us a peek into that as well as what's going on with the void. Here's Courtney Nash of the Void.

This is Textron tv. Hi everyone. Welcome back here to our next segment on Tech Drunk tv.

Yeah, I've had this, uh, lady on our tech drunk TV now, I think since she started the very first kind of void reports. It was probably three years, two and a half years ago maybe. Yeah, it was, uh, yeah, 2021 was the first one, so Yeah.

Going on. That's what I said, two and a half going on three years now. Yeah.

Let me introduce you to Courtney Nash. She is the creator of the Void, VOID. Courtney, welcome.

How are you? I'm well, Alan. It's good to see you again.

Thanks for having me back on. Good to see you. Um, so Courtney, we're gonna talk about, uh, the 2024 Void Report.

Yes. Exploring unintended consequences of automation and software. But before we do, I think we gotta lay a little bit of groundwork on kind of what the voids mission is.

Yes. Right. What you guys have been doing for three years and how it relates now to this 2024 report.

Maybe a little bit about the Void as a business too, right? Does the community, and there's, there's other services and stuff tied to it. And while we're at it, let's talk a little bit about you if you're not shy about it and, and coming not shy as the, I'm not shy.

Uh, you know, what, what, what role you're playing here as the creator and kind of curator driver of, of the community and of the whole project. Yeah, I mean, I gave you a lot. That's a lot.

I can run with it. The, the way I got here, the way I got to the void is a part of, you know, the whole story in that I, um, well, I mean, you know, I, I'm Gen XI grew up with the internet, right? Like first generation, but it, it, it's turned into something very different than maybe what we'd all imagined, um, it to be.

And I eventually started the void because I'd spent a lot of time at O'Reilly Media, uh, running conferences, talking to people in tech, looking at where things were going. And I, I have a background in social sciences and neuroscience, and, and so I wanted to bring a lot of that research, um, knowledge to the technology space, and especially in human factors, the roles of humans in technology. And so when I started the void, it was because there was no source of information about when things go wrong, right?

It was software. I mean, there's lots of sources, if you will. There's everyone's individual sort of knowledge and experience.

All of these learnings that people have as engineers that they take with them as, right. They move around or, you know, organization, organization. But we didn't have sort of a central repository to keep track of incidents, right?

Of software incidents. And that has to some degree existed in other industries. So looking at other industries that have studied how things go wrong, right?

Like airline aviation, nuclear power plants, like, there's a whole wealth of research and information that people have used to try to make those industries safer. And there's a bunch of us that care about making sure that this internet thing that runs our lives now, um, it, we also have information about what happens when things go wrong, that we're all communicating with each other, sharing that information, sharing that knowledge, right? So the Void was started as a database of incident reports.

So I'm not going and, you know, investigating people's incidents, but when people write them up, we bring them into the void as much as we can. Um, and we love it for people to submit their incidents. And you can do that at the Void community.

Um, there's a form there to, to share any of your incident reports. And so over the last two and a half, three years we've been building this corpus, and in part, trying to use that information and those data to bust some myths, to be honest, we have a lot of assumptions, I think, about how software works, about how software fails. And so we did that a couple years ago.

We started looking at things like MTTR, um, and the notion of like root causes and software. And so we, we, we chased some of those myths. I think we did a pretty good job of using the data in the void to bust some of those myths or to at least change people's mindsets, right?

About how software, how this complex system works. So this year we wanted to look at automation, as you can imagine, why we care about automation right now, um, at this like point in time, right? Um, well, it, it's the dawn of an error perhaps, right?

Where automation is going to Solve all our problems, Alan, it's gonna solve all Our problems. Yeah, of course it does. Yeah.

Right? So I'm not Gen X, I'm, I'm the end of the boomers. I've heard that before.

Um, right. Yeah. But go ahead.

So, so that was this year's focus, um, because, uh, it just felt like the right timeliness. And I had, after talking to a lot of folks, I had this realization that we had a different type of data in the void. So previously we chased things like people might give us information about how long the incident is right?

Or how severe they felt it was, and those kinds of things. More quantitative data. But we have this wealth of qualitative data of text-based data, right?

You all, everyone who writes one of these spends a lot of time describing this and, and writing about what happened. And so we took some cues from social sciences, um, and did what's called like a thematic analysis. The details don't really matter, but we delved into hundreds and hundreds of incidents in the void and looked at the ways in which automation pops up in software incidents.

Um, because the general law is right, we'll automate the detection of problems. We'll, we'll automate the resolution of problems, we'll free the people up to worry about the bigger things. And that's really not actually what happens.

Um, so all these other domains have a lot of research showing how when you add automation to a complex system, it can make things harder for the humans. So if you've ever talked to an airline pilot, you know, there's, there's, everything's fine until it's not, and then you don't understand what's happening, right? So debugging a situation when automation's been running the show is where the, the real, um, difficulties start to arise.

And we found things like 75% of the time people still have to jump in and fix it, right? It doesn't, it doesn't, it doesn't auto resolve itself. Automation plays all these different roles in software incidents.

It can detect it, it can be one of the, you know, contributing factors. It can maybe solve it, it can make it worse. Mm-Hmm.

And a lot of the time we add more automation as part of the fix, right? Right. So we're layering like, you know, all of this, you know, and every time you add to that, you add to the complexity.

So we really wanted to, to show the, the, the tapestry of, of how automation is involved, because I think the, the assumptions around it that we hold as, as humans, um, and then the, the perhaps some of the lore in marketing in our industry is very simplistic about how automation can be involved. Well, I think it appeals at a base level to people. Yeah.

Right? The auto home automation thing. Yeah.

The, the general thing is computers are good at blah and humans are good at other things, right? So we'll just let the computers do these things and we'll let the humans do these things. But that's, that, that's not Works.

But I think that conflicts with this dawn of this new era, right? With AI and everything. You know, one of the, one of the words or phrases that I've learned in this AI world is unstructured data.

We, we have to collect this unstructured data and, and bring and bring re and bring, you know, order to chaos. And, um, I mean, and look, when we look at what the void collects, we're talk about unstructured data there. That's A lot of unstructured data.

There's a lot of Unstructured data there. And, you know, how do we go? You wanna, Somebody asked me on a webinar yesterday, why didn't you just train a large an LLM An LM with all this?

I struggle, throw it all in a vector database. And the answer Is because put shot in front of it, Humans have expertise that we have developed and acquired over time, right? So I had five, 600 incidents that by a keyword search through these unstructured data, told me that automation was involved, right?

And I could have just left it at that. I could have done some keyword searches and wrapped it all up, but I had to go in and look at them. I had to read them.

Some of them were, you have an automation product and your automation product failed, but we don't know if it failed because of automation or not. But that's not what I'm talking about. And you know, someone could say, well, eventually an LLM we'll figure that out.

And which is a lot of hand wavy, you know, we don't know. But I, as an expert, had to go in and read 500 and something incident reports to make those kinds of detailed decisions. Like, yes, these ones are, should be included in the analysis.

And then I had to read those other ones much more closely, and I ran a little bit of that stuff through, let's just say chat GPT. Okay? I am not, you know, and yeah, it's not, it wasn't even close.

It was not the same. Um, and so it's that role of expertise that humans have in, in being close to those systems that makes the difference in understanding what's really happening. And especially when things go wrong.

Um, and, and we put automation in to make it easier, but when the humans aren't close to that, right? They can't see what's happening. It's harder when it stops working.

Um, and so it's, it's a, it's a paradox. Um, and then actually, I, I talked about some of the research. There's two different great papers.

I love one's called the ironies of automation, um, uh, by Lisse Bainbridge. That's not even anything to do with software, really. It's back from 87, I think, um, in, you know, sort of other kinds of systems.

And then the other one is automation surprises, um, which is a little bit more modern research. But these, the automation is ironic. It is surprising.

It doesn't do what we expect despite the fact that supposedly, right, we told it exactly what to do. The systems are, they're too complex. Um, and, and the Mythology Around it.

Yep. I I think you just hit on it, right? When we have complex systems, it's, it's much harder to automate them.

You need that human, I don't know whether you want to call it in intuition. Expertise. Yes.

Expertise. Um, and look, I I'm not sitting here saying that as AI continues to become smarter, if that's a word we could use that, it might take over more of those things. But you know, Why, why are we trying to replace ourselves, Alan?

Like, why did we call it artificial? Oh, Because, so I, so that's a great question, but that's tied directly into the automation thing, right? Yeah.

com March 14th, it'll be 10 years. One of the things, like one of the bedrock sort of underlying thesis, automate all the things, right? Um, is, is humans will never be able to do it fast enough as the machines keep going faster.

And so we need to automate the humans out of the process of we're gonna do it faster with high and, and still have higher quality. Mind you, I'm talking about software now, but, um, it's, it's what people think. Yeah.

And, and so that's, we're trying to peel back the layers of that and show you how those assumptions can fall apart or can, can lead to, in particular, the assumptions lead to the way we design these systems. Mm-Hmm. Like, we could think about it, like, the nerdy phrase I use in the report is joint cognitive systems, which comes from, you know, the human factors world.

But, um, that instead of, oh, we'll replace us and we'll just get the computers to do things fast. Why aren't we thinking about this as we're working together to do this work? Right?

We're trying, we, you can't do just one or the other at this point, the way we've built it. Um, but it's, you know, it's augmentation and it's helping humans do those things better versus, but you know, it's, we're we're placing auto automation into a very, you know, profit driven mentality too, right? We're gonna do it faster, we're gonna do it cheaper, we're gonna do it.

Uh, and, and eventually that short term view of things leads to these longer term brittle scenarios Where we Absolutely. But that, but you just hit on another sort of principle of, of the market, which is our most expensive resource many times, is the, is the person behind the keyboard Yeah. Is the Brain.

That's because like the majority of the time, we have to still just step in to solve it. You can't get rid of us. Yep.

And I, and I think it's gonna be that way for a while, but that doesn't stop us from trying to say, you know, this is what we need to do. We need to get less humans in it and go faster. Yeah.

So, um, And uh, it's it that's your job to combat this and bring, I'm trying to counter that, right? Like bring, Do you really to the Darkness a robot flying your plane? Of course you don't, you know what?

But I think a lot of people say in five years, I will, Oh, it'll somehow it'll be safer, or Yeah, well, it's the, well, but I, it's Standing of, of autonomous driving, right? It's a car instead of A place. But we're seeing that all apart right now, right?

Um, I wouldn't say as an industry falling apart, but I would say we're seeing the assumptions crumble a bit and the reality emerge from the, the hype around that right. Of What's actually possible. But I think most people say, most people say, well, we just haven't perfected it yet.

We gotta get it better. But, but let's take planes for instance. Right?

Planes are probably the safest mode of transportation we've ever had. Yeah. And, and because a number of things though, right?

Because a while back when it wasn't safe, everyone got together and was like, okay, what's going on? We're all gonna share a bunch of information. We're gonna be very open with each other, but heavily regulated, which we don't want.

Um, and, uh, a huge amount of research into how to make those environments safer, which we also don't seem to really wanna invest in, which is what we're here for, right? We wanna bring that kind of research to our industry, um, because absolutely the internet runs everything now. And, and, and that be things that we didn't think of as safety critical are going to become safety critical.

Um, you know, when with Great power comes great responsibility, You don't even realize like local governments and other countries were building on top of like, you know, Facebook chat. And so then when Facebook went down a couple years ago, you literally had like critical, What about just yesterday at t went down 9 1 1 service went down with it, people were freaking out. Yeah.

You know, the rumors started. It was a nation state attack, preclude prelude to war and everything else, But we have to take these systems we're building. That's the world we live in.

Yeah. You know. Hey, Courtney, we're almost outta time for people who a just want to even not even looking at the report right now.

Yeah. People who wanna maybe join the void, or, I mean the void, the com the void community, they can, I obviously joined for Free's, but there's also Two aspects. Yeah.

So there's what we're calling the community. You go to the Void Doc community, you can see, you can get the report, you can see the database, we, you know, all the things. We've got a newsletter, we've got a podcast, we're doing all the things.

Then there's the other side of it, which is trying to help really bring that information more purposefully into organizations that want that help. So we have the Void Insights, which is an enterprise subscription model. We have a bunch of really cool companies in there already.

Um, and a little bit of space left for this year. We have, um, sort of, we bring in experts, so researchers from the field, lots of people I mentioned in the report. We have joint like learning sessions together for, and people sort of chat and house rules.

So people from pretty amazing organizations getting together and talking about the research, the incidents, what's happening with them. And we have, you know, webinars and a bunch of other kinds of things. We even have, you know, if people want Individual One-on-one, you know, sort of advice and consulting for how to work on these topics within their organizations.

That's what The Void Insights is about. And people can find out more about that from the Void community as well. Excellent.

I love it. And the report's there too. Courtney, thank you so much.

I, you know, we don't talk enough, but every time we do, I learn I'm not putting out more than one report a year, Alan. So we gotta find some other ways To get, get together. Alright.

Alright. Look, I can ask anyway, keep up the great work at the Void. We're watching and reading and learning, and thank you for all you do.

Thanks Lan. Appreciate it. All right.

Courtney Nash, creator of the Void here on Tech Trunk tv. We're taking a break. We'll be back in a minute.