AI Governance Challenges – Techstrong AI Podcast EP51

Hello, and welcome to the Techstrong AI Podcast. I'm Amanda Ani. I'm excited to be here today with Greg Whalen.

He is the CTO of Prove ai. How are you doing today? I'm great.

How are you? Thanks for having me. Happy to have you on the show.

Can you share a little bit about prove AI and then some of your experience? Yes, absolutely. So, at Prove ai, our mission is to de-risk the development of ai, right?

And specifically right now, we mean generative AI and LLMs, right? So the, the problem we're reacting to, um, you know, that we're hearing from customers, right? Is that, um, there's a lot of organizations are just getting started with, with ai, right?

Developments, they're starting to figure out, um, okay, they built something useful. Now let's start putting this into production, right? Let's start using this thing, right?

Like, like any other sort of software in the past, this raises questions, right? And, um, since AI is a very different animal, AI development is a different animal than traditional software. What, what, what we're seeing is that traditional approaches to risk management ends to, right?

The, you know, the types of things you need to worry about when you deploy software are different, right? And they're starting to confuse people and, uh, right, they're starting to put on, you know, the breaks, right? So at prove ai, what we're, what our mission here is to say, let's not repeat the mistakes of the past, right?

So when computing paradigms have changed and we found out more productive ways or faster ways to develop software, right? Let's not make the same mistakes and, uh, just continue as ignore the problems, right? And then find out later on that uhoh, we're gonna have to redo everything, right?

We have to reinvestigate our approach to risk management in this that, right? Because it just slows everything down. People get annoyed, and, you know, those who are better prepared for it will leapfrog and be able to be the market leaders, right?

First in market. So, right. Our software is specifically designed to help customers through those types of, um, challenges by basically preparing themselves more long term for doing AI developments, um, you know, at scale, which we think everybody's gonna have to do, um, given the importance of it.

Absolutely. And I imagine, you know, having to backtrack, not only is it a waste of time and energy, but probably a financial strain as well. Absolutely.

Absolutely. So, so yes, there's, there's immediate cost ROI here that comes down to time and productivity, right? So in addition to, to mitigating risks and things like that, right?

Even if you sort of say, well, we don't know what the risks are yet because we're still so early in the journey Yeah. That that's true. But imagine the cost, the financial costs you're going to have by, by basically furloughing, you know, many developers for, and, and introducing processes that worked for cloud, you know, cloud development or web two development that don't particularly work well for AI development, right?

The costs are very big. By then, you'll have teams of people working on this, probably relatively highly compensated teams, right? And basically by, by making it such that they can't spend every waking day developing software is lost money, right?

And, you know, we think that there's returns by investing in this now, so you never have to get into that state to begin with. So from your experience working with different clients, what are some of the biggest risks or problems that you've seen them encounter? And what advice do you have to avoid those risks?

Yeah, so think that the, the biggest, the biggest risk or the biggest, uh, you know, problem that we can solve at the high level, it's a business problem. It's not really a risk, it's, it's, it's understanding like, and communicating what our mutual obligations are between multiple departments or multiple, you know, parties. So it's the breakdown in communication between those multiple parties.

That's the biggest risk. So it's like, I, I say that for instance, let's take a very practical example around data assets and data usage, right? So me as a vendor, I have a solution.

It maybe it's a, maybe it's a great solution and, uh, maybe it uses, um, some sort of AI behind the scenes. Um, you know, and, and maybe I wanna be sure to somebody who buys the solution around what sort of data assets it's using right behind the scenes. So I want assurance that every time that you change something, you're gonna guarantee to me that you're, you're, you're applying the same ethical standards, the same kind of legal standards that, that I need to make sure that I can keep using, you know, your software, because I don't know what sort of training data is actually being used behind the scenes for your solution, right?

So you have a very practical problem in the old world, right? We would've maybe just signed some sort of contract or something like that, that says, Hey, you're not gonna do thing X that I don't want you to do. Or you would, you would establish an SLA that establishes uptime and availability and error rates and establishes what we're going to do when we breach those types of things.

Uh, the, the problem in today's world is that doesn't, the problem in the AI world is that doesn't work particularly well anymore, right? Even because the, the sense of what's, what's good in what's bad or what, you know, what you can do, and what you can't do is so blurry that unless you're really in the details, right down to the level of exactly what data is being used, by whom, by where, where is the model running then, then the whole, the whole discussion around mutual obligations is pointless to begin with because it's, it's impossible to write in just standard right? Contractual terms anymore.

So, so like, you know, the, the biggest like, problem number one is simply expressing, right, what we actually expect of each other. And, um, for proof ai, right? We do this in the form of modules, software modules, right?

That allow you to concisely and clearly in, in code, of course, right? So this is this, it still has to be translated to code. Like, you know, let's get into the details of your data sets, which detail, which data sets are we talking about?

Which models are we talking about specifically who owns them, right? And now let's, let's track the events from those particular ido, from the systems that manage those. And let's, let's talk about and define what's good and what's bad, right?

What's acceptable, what's not acceptable. And when things start to go un unacceptable, I wanna be notified in real time as a customer that hey, we're in a potential breach so that we don't have to have a big complicated, you know, discussion about it, right? Did we or did we not?

Right? It's, it's simply a, this was the model we were talking about. These were the data sets we were talking about.

This is what we said we would do on update. Are we doing those things? Are we not?

Right? It's a much more practical discussion, right. To have than than others.

So I'll answer it that way, right? I think the biggest discussion is there, there's sort of this fear of, okay, I wanna use this solution, but I really have no idea what it's doing behind the scenes. And then, okay, I I, I want to address that problem, right?

I wanna have a discussion about it, but the discussion is unruly because it's so difficult to be specific, right? And, and, and, um, you know, that's actually the, the first problem that we think is gonna, we'd like to solve, because that's what gets in the way of, of doing anything right now. It's just, uh, you know, I'm, I'm a customer, I'm worried I want assurances.

Um, I know what assurances I want at a high level. I don't know how to define it and I dunno how to track it. So, you know, you can start there.

That's one of the first, you know, problems to, you know, to start with. Absolutely. And so I know there's a lot of, um, new regulations and AI laws, uh, in fact and few introduced by the eu.

Uh, how do companies need to, um, address and, and, um, adhere to these new laws and regulations and stay on top of those? Yep. Yep.

So, um, we think that at least my, my personal take is somebody with a technical background. Yeah. Right?

So I don't have a compliance background, right? But, but of course, over the, the past many decades, right? Have, having been through many compliance waves, right?

I think that the best way to stay on top of things is to, you know, approach these things. As you know, compliance is basically the bare minimum, right? So like, whatever's going on in compliance is very, very much worth your time to look at today, right?

Figure out, hey, you know, if, if I had to comply with this tomorrow, could I, because even if you don't, it's still a very good thought exercise, right? Around where would I have to start? Because generally, compliances are not, you know, they don't fully tell you how to manage risk in your organization, right?

They're very much a sort of baseline standard about, Hey, let's be sensible. You should at least be doing these things, right? So in general, my my take is that you, you don't wanna be caught sideways in another year when you find out you have to comply with something, right?

That's the, that's the way that people have often, you know, said, oh, we don't need to worry about that IO standard yet, because it's not a law, right? So let's just wait. Um, okay.

But then when something does happen, even if it's not ISO, it's something similar to an IO standard, and then you probably could have been, you probably would've saved yourself a lot of pain if you had just said, yeah, let's, let's read the standard and let's, let's make sure we could roughly comply with it now, because that's gonna probably be a pretty good, you know, framework and platform to start with, to actually manage risk in my ai. So you, in summary, I, I think it's, uh, you, regardless of when they're going to become required or whether they're considered sufficient to transmit to other parties that, Hey, I'm doing what's, what's legally required, even if that never happens, the point is, a lot of people started and put a lot of smart people, put a lot of thought into the AI ISO standards that really it's worth looking at and saying, maybe I should just start there, and maybe that's a good, you know, starting place. And then let's build on it to figure out like, what do I have to do extra to really manage, you know, risk of AI development either with my partners or internally.

So still very much recommended that you start, you know, looking at it as a baseline just because, you know, where else are you going to get an, you know, an output from many people who are very experienced in the, you know, in the field telling you that, Hey, this is probably a pretty common thing everybody should be doing, right? It's, it's, it's worth doing now so you're not caught unaware in another year. Right?

And, you know, even if it's not law, somebody's gonna ask you those same questions because they're very much a baseline, right? None of these things are outlandish, crazy things to do. They're, they're, they're very much a, a, a, a bare minimum in many cases, right?

So why not, why not start? Now? I understand many business leaders have concern with budgets when it comes to integrating new AI tools, uh, to solve problems, and they're looking for that return on investment.

So what advice do you have for them? Yeah. Do, I guess, would you mean for like a compliance tool or actually just for an investment in ai, It investment in AI or compliance tools?

Um, I'd like your opinion on it. Sure, Sure. So for, I mean, I, I think most of the investment, we, at least from a compliance tool, right?

We wanna see that the investment is tied to the actual solution, right? So, right. That they believe enough in AI that in their AI project, that they know exactly what they're going to, you know, get out of it.

So like, if you're in, if it's a, if it's a relatively common customer support use case, right? We're talking about having a very clear idea that, you know, we want, you know, you know, we wanna automate ticket response, we wanna see, you know, customer happiness increase, we wanna see throughput, and we ultimately want to lower, you know, cost of, um, you know, cost of, you know, cost of serving costs, right? The cost to serve the customers.

Um, I mean, that would be an example. So I, I don't, so I mean, the recommendation would be that, you know, rather than thinking about investment in compliance is separate, I think it's, it's absolutely necessary to just think about what is the actual, you know, return on this, on the AI thing that you're, the ai, ai embedded product that you're working on. Um, so it's gonna vary wildly by, uh, by use case, right?

Customer support is usually error rates decrease, right? Customer, uh, you know, time to serve customers drops, right? Which is good, right?

And the error rates, error rates lower and overall lower labor costs, right? Decrease. Um, you know, for other things, let's take, you know, the ad tech space, right?

And targeting, right? Hey, targeting, you know, microsegmentation gets even more precise, right? You get to, you know, basically, you know, the, the revenue per customer, rev revenue per person serves, right?

Can increase in things like that, right? So it, it varies wildly, right? But, but usually you're talking about in all of these cases, right?

Doing something more quickly than a human being could do. So typically when you're talking about AI, is you're talking about doing a task that was traditionally assigned to a human right, was relatively slow, was a relatively, you know, large pain point, right? And now you're going to eliminate that sort of manual review, uh, you know, not because the human was slow, it just wasn't a task that was, that was scalable towards, you know, towards, um, you know, a human line of thought.

So we're saying, you know, we got software up to a point where it was automated, there was a manual check, typically, typically an AI of some sort is replacing, you know, what used to fall back on people, right? So you can imagine the savings is always, you know, better accuracy, you know, at lower cost. Um, so what you then, the return though of a compliance tool like us is to say, okay, but, but, you know, there's a hidden cost here.

So I think that's a, that's a big cautionary tale here to people deploying ai, is that, um, okay, yes, you're gonna save this money if this AI performs well, unlike a human, right? AI don't always have, or AI agents or, or whatever, they don't always have co, you know, common sense, what we perceive as common sense, right? Which means that, you know, you, you might have a very anomalous, unexpected error that has a huge blast radius, right?

That's just like, whoa, like a human would've never made this error. So the AI may be like, wow, this thing is, is awesome. It really does in fact, perform better than human review.

It does a lot faster. But you're going to find these unexpected untestable, undiscoverable, you know, kind of, um, instances where there's just a big problem. It's a high cost, right?

High brand damage, right? It could be a, it could be something as saying, you know, of, of a, a poor refund handling or something like that, where, hey, there's an exploit and, and suddenly your automated solution is starting to give out refunds that really would never have ever happened under human review, right? So there's monetary loss as well, and we've, we've seen actually, you know, some of these things have actually already happened.

Um, you know, so, so we say, okay, well then the return on investment of a compliance tool or of a risk management tool de-risking AI, is to say, yeah, those things are going to happen, right? And right, we're going to prove return on our investment by saying that when they happen, the blast radius is going to be as minimal as possible. The o the downtime is gonna be as minimal as possible.

This is gonna be traceable, right? It's gonna be very clear who messed up, right? So a lot of companies don't have that today.

It's gonna be a discussion when something happens and it will happen. Uh, it's going to be unexpectedly large in some cases, right? The impact is gonna be, whoa, you know, how did that happen?

Uh, that's very damaging. And then it's gonna be who's at fault, right? How do we fix it?

Right? What actually happened and who's at fault? Um, without having something in place that actually tracked every single thing, every single prediction, and every single stage in the development across multiple organizations, right?

Multiple data sets, multiple parties, then you're never gonna be able to get to the bottom of that question, right? It's gonna be a finger pointing match, it's going to be a, well, we need a couple of weeks to dig into the logs and then share logs with each other, right? Which is a very traditional software development process, right?

There was an outage. I want a root cause analysis. I want an action list of what's you know, of, of how to, how this, uh, how we can never repeat this again.

It's not gonna work very well with an AI model because there's no specific list of actions, right? Other than we need to review things manually that you're gonna come up with. And we've already said earlier that that's the wrong approach, right?

Because that doesn't scale either. So, right? That's what we think the return is.

The the return here is, um, is very much around managing those, those sort of big blast radius anomalies that are inevitable, you know, going to come up and then reducing the amount of time to resolution, reducing the amount of time and effort spent figuring out who did what, right? And then figuring out how we never happen again at a precise level, right? Where in the chain did this actually fail?

So a very helpful tool to have, especially as we see AI being utilized for so many more things and so many more automations. Absolutely. Yes.

All right. Well, if there was one key takeaway you could leave our audience with today, what would that be? Yeah, so going back to the beginning, I, the key takeaway I think is to get started with, you know, de-risking now, right?

So obviously that's, uh, a bit self-serving as you can imagine. But I think it's, uh, it's been the right advice, you know, to customers over the many waves of computing, right? It's, it's, as we started to do cloud development as we did web development, right?

Getting, getting ahead of it is going to save your developers and your organization a massive headache by doing just a little bit now so that you're prepared when inevitably a customer asks for something, right? They say, all right, we, we need to have a risk discussion. Right?

Or a regulation is imposed and you have to go react to it. So, you know, the, the, the, you know, the recommendation here is you're gonna need to move fast, right? Don't wait.

Um, thinking about compliance and risk is actually pretty healthy for an organization now because it allows you to move faster, right? You don't have confusing discussions with leaders and pe right? It forces them to start to get educated now, which is actually better, right?

So to your earlier question about return on investment, right? It's, it's probably net positive, right? Even if you don't have a lot of AI now, because it, at least it brings everybody to the table to talk in a data-driven way about risks and about specifics of a model, partic, you know, potential pitfalls, right?

In a way that really we've seen a lot of customers have challenges with. Alright, well thank you so much for coming on the show and sharing your insights with us today. Uh, Thank you for your time.

I appreciate it. And thank you to our audience. Stay tuned.

There's more.