Rich Waldron: Turning AI Agents Into Measurable Business Value

Hello, and welcome to the latest edition of the Techstrong AI Leadership Insight series. I'm your host, Mike Bor. Today we're with Rich Waldron, who's CEO for trade, do ai, and we're having a chat about, well, AI agents and how to track their performance and measure their value.

Rich, welcome to show. Thanks, Mike. How you doing?

I'm well, I'm well. I think everybody went from what is an AI agent to pretty quickly saying they're gonna be billions of them, and I'm not quite clear how we're gonna afford all that, but, uh, at the very least, maybe we're each gonna have a dozen of these to manage and try to figure out. But, you know, there'll be dozens and dozens of these over the spread of an organization, and then it becomes, well, how do I know which of these AI agencies doing the right thing, performing well?

And for that matter, could wind up being more trouble than it's worth, but how are we gonna evolve from here? So, yeah, I think it's a, it's a interesting question because you are totally right in the speed at which people went from, Hmm, I think I might need an agent to, I now have a lot of agents, what do I do about them and how do I figure out, uh, how effective they're being? I think there are kind of two ways to look at impact or effectiveness, and often they get conflated.

So on one hand there's the kind of technical side of it, which is, is the agent doing the thing that it should be? Like, is it technically sound? Is it carrying out the actions in a verifiable or accurate way?

Can I determine that the sort of implementation or the execution of this agent is, uh, is up to par? And then the second question is, is it actually doing anything valuable? And that, that latter piece is far more around figuring out what the impact of the agent is, and that that's not dissimilar to kind of taking on any sort of consultant or thinking about any line of work that we do today.

In that you sort of start with, here's the thing that I'm trying to solve, uh, what will determine whether or not I've been successful. And so I think there's, there's sort of two ways to kind of look at the effectiveness of a, of an agent. One is technical execution, and the other is, uh, overall impact or business impact that it has.

Mm-hmm. When I talked to folks, at least early adopters right now, there seem to be in some sort of mindset that says, well, the AI agent never quite does the same thing the same way twice. So they ask it to do something about four or five different times, and then they pick the one outcome that they seem to think is the best suited for their task.

But that seems kinda an expensive way of going about doing that, shall we say, because now I am, I don't know, let's say I'm, I'm throwing out 80% of the output that I created for the 20% that I have preferred, and ultimately the cost of that gets a little crazy. So how do I kind of like start thinking or putting some sort of, um, controls or governances around that, that activity in a way that won't break the bank? Yeah.

So I think that falls into the first camp that I was describing, which is the evaluation of the technical implementation and the way in which it, it's executing as a, as a function. And I think really the, the challenge here for a lot of folks is that we used to kind of buying software or interacting with software in a way where once it's set up, it kind of just works and it often stays the way that it, you know, we implemented it, um, until further down the line, something else in the business changes. Whereas an agent kind of needs constant nurturing.

You know, if you think of a agent, much like, uh, the, the analogy I use is it's the, it's the intern that kind of knows everything, but what the intern doesn't have is necessarily the experience. That's the thing that you are, you are implementing or, or kind of educating that, that intern on over time, which is how they ultimately become more effective. And so there are, there are a couple of sort of basic, um, actions that I think people need to take.

The first one is really the feedback loop. Uh, and this is, you know, probably the lowest hanging fruit. And one of the most important things, which is to handle the thing that you just described, really, you're trying to get a feedback loop going whereby you are indicating back to the agent whether or not it handled the, uh, question or the action in the way that, that you wanted it to.

And that feedback loop then kind of feeding that insight back into its memory, meaning that the next time around it's gonna perform in the, in the way that you want it to. And so that, that kind of loop of, um, helping to sort of train and, and, and push the agent in the right direction is necessary. And the speed at which you can make these agents more productive, um, is partly through a feedback loop.

And then it's secondarily through figuring out what scope you are providing the agent in the first place. So how these things end up being expensive is if you're providing too broader scope, uh, and expecting an agent to carry out a much wider range of, um, uh, actions. Therefore it has a much bigger aperture to, to kind of, you know, get wrong or, or not, not act in the way that you like.

So keeping that relatively limited scope, introducing those feedback loops is the fastest way to getting a, an effective agent from a technical standpoint. Yeah. So to your point, we've established that there's cost to these things and we can put some controls in and maybe best practices to minimize those costs, but there's a difference between cost and value.

So how do I understand what the value of the AI agent was to the business? Yeah, and I think that the, the value piece is this is something that, um, uh, you know, we actually have a lot of experience at. You know, there, there hasn't been a, uh, a software project that I've ever worked on, or at least a good one that didn't have some semblance of how we measure ROI at the end of it.

And I think the, the agent implementations are very much the same. So a a bit of a classic example is if you think of, um, using a support agent of some kind, maybe it's a IT support agent or a, or a customer support agent, the way in which you're measuring impact is, um, through things like number of tickets, deflective or, or responses handled. And then you're effectively, you know, you are then being able to measure that against what your expectation is, um, uh, prior to the agent being around.

And what did that allow you to accomplish that you, you weren't able to accomplish before once you start getting some of these business metrics in place? You know, another example being, um, we've worked with a, a, an organization that's implemented a, an agent that basically does call preparations for, um, account managers before they get on a customer call. And what they discovered is account managers in general, were spending between 45 minutes and an hour sort of researching an account before going and, and getting on the call.

And they were often going to the same places, getting the same sources, setting up in the same way. So they, they were able to put some simple time tracking in place to figure out, okay, well how much time is this agent saving each, you know, account rep, uh, uh, per week? And then you can figure out, well, what does that look like over the entire workforce or over the entire group?

And then you'll start to get to a place where you can figure out, okay, well how effective is this agent being, like, what, what impact is it happening, uh, on, on the bottom line? And then ultimately the top line? Because in theory, you're in a better position to support your customer.

So I think that that business outcome angle, um, is a little bit tangible and, and kind of based on, uh, each individual business, but we, we should be thinking about this in the same way that we would in with any sort of project that, that people are embarking on. Mm-hmm. There's a lot of debate about, um, how best to pay for the AI agent.

And some people are saying, you know, it's token, others are per seat and others are length of time of usage, et cetera, et cetera. Um, but the value of the AI agent will differ widely per business and per task. And so can we rethink how we're gonna price the way we consume AI agents?

Is there another way to think about this? I think in, uh, you know, what 25 years of SaaS delivered software and, and many years before that of, uh, on-premise and ERP delivering software, I think the, the pricing model has kind of constantly evolved. You know, we, I think e even SaaS itself, it was pretty standard to charge per seat.

Then consumption started to be introduced because the workloads or the payloads began to change. And I think with ai, it, it, we're gonna see a similar sort of oscillation between pricing models. I think quite early on there was a lot of discussion around outcome-based pricing, but to your point, everybody's outcomes look slightly different.

Um, the value of the outcome is slightly different. You know, if you've, if you've got an agent working on a, uh, account that's worth a million dollars or, um, a hundred accounts that are worth $10, like how do you think about the effort required or the way in which you sort of break that down to be able to develop a pricing model on a, on an individual basis? I think what I'm commonly seeing right now is more of a sort of consumption slash token based approach, which companies are then using to get a sense of what their normal course and speed looks like.

And then, then you're able to start trying to, you know, push that into, um, uh, models or, or at least ways of thinking about pricing that are slightly more applicable to the, to the action that's being taken. Um, I think long story short, I don't think we're quite far off enough along yet where, um, uh, enough of these have been stood up at a level for us to be able to say, Hey, this is the de facto pricing model. And, and, and that's the way that that, that this should be oriented.

Correct me if I'm wrong, but I think I'm also starting to see early signs of what I might call an AI agent divide. And it goes something like this, people who have a lot of expertise in a space, for instance, are able to manage multiple agents performing tasks in parallel. Mm-hmm.

And they get that whole superpower kind of value. Then there's other folks who, you know, are mere mortals and they basically can't handle the cognitive load of all that parallel processing in their heads and how to bring all that together. Yep.

And they're likely to get a different value out of AI agents, and there's probably those who can maybe only handle the concept of an AI agent, you know, performing tasks in a more sequential fashion. Is that gonna impact the value that we get out of it? And therefore, you might have some organizations that are like, we're deriving an awesome amount of value based on the skills of our people versus, uh, others that might not.

Without a doubt. I think the, um, I think it goes deeper than that, which is, I think that during this sort of initial phase, there was a lot of kind of, uh, what I would describe that as throwing the agent at the problem, right? Which is an agent may not be the right solution, but people felt like that's the thing that they should be trying to, to solve whichever problem it was that that emerged.

What I've seen through the, uh, uh, customers that we have that have kind of gone a bit further down the maturity cycle, they're starting to get a bit of a balance between, um, agents that are delivered to their, uh, employees or, or kind of, or, or provided externally to, to customers. And those are built out in a more kind of traditional, um, uh, like software delivery, um, pattern. Uh, they're very thoughtful about how those are built out.

There's a, there's a lot of, um, additional modeling that's gone into figuring out what the right execution is gonna be in some cases. There, there are multiple agents that are working together so that they, they solve some of the problems that you described, which is how you handle feedback and supervision and, and, and elements such as that. But secondarily, they recognize that actually, um, some solutions need something a little bit more deterministic, and part of that flow will actually be agentic.

So that's more of like a agentic workflow model. It's not leaving everything up to open interpretation and, and, and for an agent to go and figure out. And I think at the other end of that spectrum, there are companies that have kind of said, Hey, look, we'll open up the budget and we'll open up the tools and we'll let everybody experiment, and you're gonna get a pretty mixed bag of results because in the same way, uh, your average employee perhaps isn't that great a prompting, they also don't necessarily know, you know, how to solve their own problem or, or, or, or how to construct an agent to go and do it in the first place.

So how you approach that as a company has a, um, significant impact on the overall outcome. You know, the, the technology is no doubt astounding and evolves at a rapid pace. How we harness it is really, you know, where the, where the difference is made.

It almost seems like there's two vectors to kinda master here. Um, one is, I gotta make sure that the data that I show the AI agent in the first place is of sufficient quality to drive the outcome I'm looking for. But secondarily, it seems like the AI agent more so than the copilot, needs more context.

And you hear the phrase context engineering. Mm-hmm. And much of what you just described in my mind comes under the heading of context engineering.

But yeah, there's an art to that, right? Because I also talked to other folks and they're like, you know, starting to use phrases like context rot because they gave the agent too much context and then it went off and did all kinds of things. So how do we kinda strike a balance here?

Yeah, I think a good way to think about an agent is, an agent is as effective as the knowledge you give it and the, uh, tools that it has available to itself. So if, uh, if, if you know, AI's uh, overarching skill is reasoning, then the knowledge that you make available to that, um, uh, to, to the LLM to be able to reason it significantly impacts its ability to, to react or make the right decision. And then secondarily the tools that it has available limit or enable the impact that it could have.

And I think what a lot of people have discovered is that agents that have a more specific purpose are far more effective. You know, if they've got, uh, less knowledge that they're working from that is very rich and they have specific tooling which allows 'em to carry out those tasks, then they can be extremely effective. Uh, I think the, the mistake I often see is this idea of like a kind of super agent, uh, uh, whereby I'm gonna give it access to everything that it could possibly know, and I'm gonna give it as many tools as it could have.

And that has a, a kind of paralyzing impact, right? You, you, you get into that whole context rot debate and you get into really a bit of an inability to take an effective action. That's why you hear a lot about the agent to agent model, because then you have spec agents that have specific knowledge and specific skills, but they're aware of what other agents around them can do, and therefore you are kind of limiting the scope, but opening up the aperture in a way that's far more effective.

Mm-hmm. Are you at all concerned that we might therefore encounter something that feels like a little bit like the trough of disillusionment when it comes to AI agents? Because, well, while 10 to 15% of the population has the cognitive skills to master them and become super humans, the rest are gonna be, you know, are basically gonna conclude that this may be just, you know, yet again, the IT industry overhyping something and it doesn't quite deliver that value.

So we might spend a year where people are kinda like, yeah, this stuff is, you know, it's, it is mildly helpful. I, I think it very much depends on, um, how each organization approaches their utilization of ai. Um, and I think that kind of like any tech cycle, you know, you, the, the people that are the builders aren't necessarily, um, uh, always gonna be the, the consumers.

And what I, what I mean by that is it matters how you construct these agents. It matters when how you think about their effectiveness. It matters, um, how you think about, um, uh, the ongoing, not just governance of them, but ultimately how they're gonna evolve.

And that is a skill that does require, um, um, a skill set and, and it requires having people that kind of think in, in, in that manner or, or in that sort of method of delivery. I think the disillusionment that you described is ultimately when we are just opening up the technology without thinking much about the richness of knowledge or without thinking about what tooling these things are, uh, are given and expecting a huge audience just to go and be successful. I think that that might, you know, where, where I think the, um, dare I say it, um, overhype or the over excitement is, um, I think things like vibe coding are a, are a, and I'm talking in a generalist way, are a great way for people to get an idea of what's possible, but not necessarily going to lead to, you know, production quality evolved outcomes.

And so really you're gonna see the, the companies that break out are the ones that take that approach, really think about how they're gonna harness their technology, really think about doing it diligently and well, and the others that are kind of expecting to just turn this stuff on and, and it's gonna produce magic, uh, I think will be disappointed. So would organizations be well advised therefore, to create something that feels like, you know, a center of AI agent excellence, where they're gonna go and, you know, work all these issues out and then kind of teach the rest of the organization how to master it? I think most organizations already have that department, and that is the IT department, you know, and in if, if I think back to, you know, my, my years in the integration world, the best integrations were always a handhold between IT and a department.

And the department brings the, the main knowledge and the, uh, business problem and the idea of, of, of, of what is required. And the IT department has the expertise in developing solutions and knowing how to, how to utilize the technology and harness it in the right way. And so I think, uh, a lot of companies have these steering committees and they're ultimately made up of, uh, line of business expertise.

And then, you know, we're, we're hearing about kind of, you know, forward engineers or, or, or, or, um, AI engineers that can work within, within these departments. I think that's very much a, a, a model for, for a lot of people to follow, because it means that not only are you building in a way where you're thinking carefully about effectiveness and ROI, but also you've got an awareness of what else exists. Because then the power is when these agents are, you know, are aware of each other and can tap into each other's capability.

If you're building in a silo and each department's trying to do stuff on their own, you know, then you're really not gonna get the value that, that that could be available to you. Mm-hmm. So how will this all play out ultimately?

Let's assume that everybody's got a dozen AI agents and then the organization will probably have a bunch of AI agents that are assigned their particular set of tasks that they will do on behalf of everybody else, and all this stuff will need to be orchestrated somehow or other, so where does the intelligence lie for coordinating all the activity of the AI agents who are all reasoning across some set of tasks, but, um, how are they sharing their information with each other and with us? Yeah, so I think there are protocols emerging, um, that, that help with this, uh, protocols like agent to agent for, for example, uh, which is one that we at Tray have adopted. And it, it all comes down to how you think about that centralization.

Um, I, you know, I think realistically there are going to be organizations that will adopt agents from multiple vendors. So this idea that there is some central, um, orchestration that a, there is a protocol that enables agents to communicate with each other. And really it comes down to how well described are the agents and their capable actions, because that's then what is used for, for these, uh, uh, uh, for the agents to be able to figure out, okay, what is available to me?

Where do I need to go for this? How do I, um, uh, how, how do I utilize the, the skills or the capabilities of, of the agents that surround me? And so, yeah, we, we at Tray have, have been building out this model, this sort of AI orchestration platform, really with that thought line in pro, uh, in the top of our mind because it's the, it's the same thing we saw from the, uh, integration world, which is, alright, well, I've bought 3, 4, 500 applications.

Well, how the hell do I get these things to communicate with each other? How do I get some sort of centralization? What do I need to be the glue that sits between these organizations?

And that historically has always been a integration platform as a service. I think the evolution for, for, um, uh, for our, for our business is very much in that realm. And I think you'll see more and more, um, uh, vendors kind of sit up and, and approach it in a similar way.

All right, folks, you heard it here. There's no magic wand for AI agents. You still gotta do the work.

You gotta figure out how to a, teach them, train them, monitor them, check their behavior, and then figure out how to make 'em all play nice with each other. Hey Rich, thanks for being on the show. Thanks Mike.

Alright, and thank you all for watching the latest episode of the Techstrong AI Leadership Inside series. You can find this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next time.