Managing AI Agents at Scale: Insights from Digital.ai CEO Derek Holt

Hello, and welcome to the latest edition of the Techstrong Do AI Video series. I'm your host Mike Bazar. Today we're with Derek Holt, who is CEO for digital, do ai.

And we're gonna have a chat about, well, where does AG agentic AI fit in this whole spectrum of things that we've been playing around with. Derek, welcome to the show. Great to, uh, be here, Mike.

Thanks for having me. I think we rapidly went from, wow, we all gotta learn prompt engineering to now looking at all these so-called agents that are gonna do certain things for us automatically. And when I talk to folks, everybody kind of nods their head, but then it quickly, they, I get a blank stare and I think the blank stare comes back to, well, I'm not sure I understand our processes well enough to insert an AI agent.

So how do we kind of approach this when a lot of the things that we have been doing for so long in various processes or kind of road and we actually forgot how they work. It's a, it's a really, really good question. And, uh, I, it, it's one of the things we think a lot about.

I think those that have, um, uh, ironically automated sort of the old way, right? The hard coded automation of processes are often the ones that are most, uh, well positioned to take advantage of some of the AI capabilities. 'cause to your point, they've sort of documented these things through codification.

But, um, look, we, we, we are super excited about, um, I think the productivity gains that are potential here, but I also see things at, at least at this current stage where you're gonna have a lot of humans involved, uh, in supervising, uh, some of the agentic behaviors. So, uh, while, while autonomy is obviously core to the, to the, the label of what makes, you know, regular generative AI different from, let's say, agent ai, um, I really do think, um, kind of breaking these things down to more smaller chunks where things can be handled and, and, and sort of the process is well understood and, and things can be handled in a more agentic way, we will build up from there. But I think some of the, the, um, the hype around some of the broader scale, you know, does everything just give it a, a bit of a direction and it goes off and solves it, I think we're gonna, uh, ideally probably evolve to that versus that being the reality, uh, in the, in the foreseeable future.

To your point, then, how will I orchestrate all these agents, many of which that a thing we're calling an agent is actually probably 20 different agents trained for different tasks. And if I want to manage something on an end-to-end basis, I have to orchestrate that. So where will that come from?

I, I think it's, it's a, it's the question and, and make it even more complicated. How do I govern it? How do I make sure that it's high quality?

How do, I mean all the things that we've thought about in what I would consider more like traditional software that is much more declarative, you now have to deal with in a much more complicated world. And I think that the key for me is, again, always taking these bigger items and breaking them down into piece parts. In the end, I look at whether it's, you know, traditional AI or, or generative ai or gene ai, all of these things are ultimately productivity tools, right?

And, and they help us ideally, uh, increase productivity, allow us to, to work on more creative work and less repetitive work. Um, I think the really interesting thinking about this wave of AI is it lowers the bar of entry even more because you don't need to know how to write code. Typically.

You can sort of just use natural language to interact at the same time to really have legs in the enterprise. It is gonna have to be explainable, it is gonna have to have high quality measures, et cetera, before I think anybody's gonna just let it go on its own. And so really, we think about this as breaking these larger processes down into these sub subprocesses, and then identifying where AI can play a role in a safe, secure, you know, high, high quality way, um, in those lower levels.

I, I think about, um, uh, the evolutions of, uh, autonomous vehicles, albeit a bit of a different technology, uh, evolution, but like we we're gonna have fits and starts on this, right? I think it was back in 20 12, 20 13, I think we all thought we'd be not driving anymore come 2025, right? But we ran into some technology barriers, and a lot of that was around the edge cases.

A lot of that was around the fact that autonomous vehicles need to be right a hundred percent of the time, in theory, for people to, to be comfortable with them. I think you're gonna see some of the same adoption, uh, transformations as AI goes from, frankly, right now, a much better search engine. And, you know, something that helps me avoid writer's block and, and maybe helps me write code a little bit faster to like literally giving the keys, you know, keys to the, to the processes and allowing it to go end to end.

To your point, I also found myself having this conversation recently, where am I building AI agents or am I buying AI agents? And I'm probably gonna have a mix of these things, but some of these AI agents seem to be coming from Salesforce SAP Microsoft, and then in other instances, I'm gonna need to build my own AI agent that maybe knows my processes better. And how will these AI agents kind of interact, do you think?

Yeah, I, I mean, this is the thousand dollar question. I mean, ultimately, you know, we've thought about the internet as maybe one of the ultimate integration mechanisms, right? Kinda loosely coupled, uh, you know, protocols going back decades, uh, that frankly probably didn't know at the time how they were gonna be used, but it worked out pretty well.

Um, I think it gets even more challenging with, with ai, right? You have a, a, a set of discussions and concerns around data sovereignty. You gotta a a, a set of discussions around, you know, how much is this all gonna cost and who am I even paying as agents are calling other agents?

And so, again, it just to me is like a, a, a restatement of the fact that we're early days, we are now seeing the potential of what's possible, but how we get there is, uh, always gonna be, I think, more incremental than, than maybe we suspect. Now, obviously this is moving very, very quickly, but you know, the old adage, I forget, I think it was a Stanford computer science professor, I highlighted that we tend to overestimate impact of technology in the near term and underestimate it in the long term. I think we're probably right in the midst of that right now.

The other thing I'm also trying to sort in my head is, today we have MLOps where data science teams are using that process to kind of construct a model, and then when we build software, we have DevOps workflows. Um, is an AI agent a type of artifact that will be built like by a DevOps team, or is it gonna be an extension of an MLOps workflow? And how do these things come together to drive something into production?

It's a, it's a great question. ai as we all navigate this future. The, it's, it's really interesting when somebody asks the question, what's, uh, what role does AI have in, in, so the way software's developed and delivered, which is a question, you know, as the CEO of digital ai, I get a fair bit.

The answer is like multi-layered, right? On one hand, we are like everybody building out, uh, workflows in, into our existing tools, um, to, um, to, to leverage AI to make it easier to, you know, autonomously generate tests or, uh, or, uh, have intelligent pipelines that set a old hardcoded pipelines where the level of governance and, and risk management is ebbing and flowing based on, uh, uh, on some AI and machine learning views. Um, so there's the tooling piece.

There's then the next piece that you just highlighted, which is all of our customers who have traditionally been using our tools for, for building, you know, more, more declarative AppSec, for lack of better term, right? Like more traditional, uh, mobile AppSec and web services and whatnot, are now saying, how do we also add in AI into those, into those app experiences, right? And often, um, they are not building the AI themselves, but they are using models and using their own data and training it.

And so now you've got not just model ops, but data ops and DevOps all have to converge together, right? To successfully deliver, um, a, a product. And, and then you have a whole bunch of new, new questions around what does it mean to test such an application?

What does it mean to secure such an application? How do we get feedback when somebody calls the support desk and says, Hey, I I was using your app and x, y, Z happened. How do we replicate that in, in, in the development environment to be able to fix a potential bug or to at least, uh, determine whether a bug exists?

I think we're still early days in a lot of those questions when you layer in, um, uh, this dependency on, uh, on ai, not in terms of how you use AI to build the product, but embedding AI into the solution, uh, itself. So this is why we exist, right? Is to help our large scale enterprise customers figure that out.

And we've been deferring referring to it internally as sort of, uh, X ops. In other words, there's a whole bunch of these types of ops, whether it's traditional development or, or data or model, uh, or, you know, probably a long list of other things, um, that need, um, some of these be best practices that we've, we've leveraged for some time now in the more traditional development world. As we kind of think about how all these AI agents might play out together, I almost feel like we're working towards where we're adding AI to the DevOps and software engineering workflows, and then we're gonna use AI agents to build that out.

And then we're also gonna apply that to security. So in effect, are we not gonna wind up using AI agents to build AI agents? And they become even more kinda complicated with lots of little inter interdependencies.

Yeah, it, it, it, it is, uh, uh, I mean, I think this is what a lot of folks are pontificating on. And, and, and, um, you know, I, I'm, I always, I've learned a long time ago not to predict things that are 10 or 20 years out, right? So, uh, I'm, I'm more wrong than, uh, than right.

But, but I, again, I go back to what's practical in today's world, in today's environment. And, and I think the key for us is to really focusing, um, not just on what the technology could potentially do, but like, what is the business problem that we're solving? What is the business outcome, uh, that we're driving?

And then is a, is an AI based solution, or is it more of a traditional solution, uh, that that is the right way to solve that problem? So like your, your scenario is potentially a very real scenario. Um, how quickly we get to that, um, and, and does that in itself create, um, a whole bunch of value?

I, I think, again, I think we'll get there, uh, um, more incrementally. The, the old, uh, you know, the destination is, may, may or may not matter. It's just the journey that's gonna get us there.

And, um, I think I see people more kind of early, early stage just figuring out, like even things as simple as, for example, and, you know, I've had this conversation, if I can't measure how productive my development organization is today, how am I gonna compare that to a, an AI powered development organization, right? So some of these fundamental, uh, elements, when you think about what it takes, um, have I automated enough throughout the software development life cycle to take advantage of some of the productivity gains that we expect to see in certain areas, right? I make, I, I write more code, but can I test it?

Can I secure it? Can I get it into production? Or am I simply creating a bottleneck elsewhere?

Can I measure the, the, the before and the after? Can I, can I get a little bit more thoughtful about the way that, that we bring products to market and using some of the data that, uh, that is there to help us go faster when it's less risky and go a little slower when it's more risky? So I think for, for all of these, I think it's gonna drive, interestingly enough, the promise of this potential future is actually driving very near term investments and very near term transformations around some of the best practices that we've been talking about for five or 10 years as a, as a, you know, a DevOps community, right?

Using DevOps as a metaphor for all of this. We have this notion of observability in DevOps. And when I look at these AI agents, something that crosses my mind a little bit is, we all talk about hallucinations, but, um, the models themselves and the AI agents drift over time, and how do we know that they're still doing what we intended them to do if we don't have some way of observing them?

I, I, I, I, I totally agree with you and I, I think what we'll come back to is number one, um, uh, the, the term observability in the industry has been very narrow, I would say historically. Uh, mostly in production, mostly in sort of one area. I think you're gonna see the emergence of big O observability where we look not just in production, we look into development.

And then to your point, we have to look at some of the more declarative, um, uh, uh, tools. Things that, you know, when you click this button, we know exactly what happens and delineate that from some of the, the more, uh, generative of tools that, that, to your point, evolve over time. I, I also think this is part of the reason why, uh, again, um, how broad ag agentic gets, how big of a workload we hand over to the models and say, have at it.

I think we're gonna start much smaller, because to your point, it's, it's much easier to, to identify some of that, uh, uh, some of the hallucinations, some of the drift, et cetera, in smaller work items than it is to see it in the bigger. And, and, um, I, I actually think hallucinations are not being discussed enough. I i, if you, if you want me to be frank, 'cause I think it is actually, um, probably for the, for the big model creators, it's not a convenient topic to discuss because it, uh, uh, is a bit of a challenge.

But again, go back to that autonomous vehicle example. Um, we've, if I'm gonna hand over the keys to the way, you know, uh, support interacts with, with our, my customers, or I'm gonna hand it over to other, like, really mission critical things within my business, it's gotta be right a hundred percent of the time. And today it's not.

And to your point, Jim, AI and the agents themselves are probabilistic, and then they take a reasonably informed guess, but we seem to be wanting to insert them into processes that are deterministic, and they're supposed to be done the same way each time a hundred percent of the time. And no model does the same thing the same way, the same every time. We, we, we experience this ourselves, right?

Just interacting. Uh, you ask one question once and you ask it again, and you get a different answer. I, it's funny, and I forget who I should give credit to this, this is not an original thought, but somebody had, had asked a question is, is uh, language knowledge, right?

Because like in the end, what the, what the, the generative AI models that are really good at is the language piece predicting the next letter, the next word, et cetera. Um, but, but even the computer scientists that have created these models have a hard time explaining exactly how it all, uh, how it all actually works. And so, um, does it really equate to knowledge and, and your points to write?

One is you get to these more complicated scenarios, um, that is where, um, knowledge, uh, is required. And, and go all the way back to your first question. Um, if I can't explain a process or how it's supposed to work, or I've not documented, or I've not codified it, how can we expect the, the, um, uh, the agents, uh, to follow it?

So again, what, what, what I like about it, it's very, very similar. If you go back, you know, the per the early days of personal computer, the early days of the internet, we, we both presented a much more robust, uh, much faster transformation than ended up happening. We also had the tech optimist and the tech pessimist at the time, right?

Some saying that all the jobs were gonna go away and others saying, yeah, but don't, don't you think we'll create a whole bunch of new opportunities for new products, new jobs, new processes? Um, I tend to be in the optimist, uh, uh, camp, um, as just seeing the last few waves. Um, but, but again, we've sort of always, I think, uh, uh, overestimated what, what's gonna happen today, tomorrow, six months from now.

And, and again, probably underestimated what happens 10, 20, 30 years from now. In general. I kind of feel like we're all a little conflicted.

On the one hand, there's been a lot of talk about eliminating a lot of the regulatory oversight around AI as of late, and then at the same breath people are saying, well, I don't know what I wanna automate using this stuff 'cause I'm afraid I don't know what's gonna be regulated tomorrow and if I'm gonna roll it back. So are we kind of stuck in some sort of quandary? Yeah, I think a little bit, although again, I can always go back to the, the history in these things and whether it's like way back to the automotive revolution to some of the other ones that had transpired, including the internet.

Um, we've, we've always had sort of these fits and starts. I think, I think a couple of things have held true. And, and, and you know, this may be ultimately different, you won't know until sort of hindsight, but typically, uh, these technological revolutions have been really around improving productivity, right?

They've ultimately about, uh, um, improving productivity and do they destroy some jobs? Yes. But historically they've created more jobs than they've destroyed, right?

Particularly in the US economy, if you look at sort of the forward leaning kind of creative destruction economy, that that has been the, the United States, at least over the last a hundred and twenty five, a hundred and fifty years. Um, I, so I, I ultimately think, you know, regulation, I'm not a, a government official. And, and there is this weird, there's this balance that needs to be striked around encouraging innovation and not pulling it back as well as, um, uh, making sure that their safety.

I also am often reading lines, uh, or, or feedback from different companies. And you gotta think about from each company, like what their incentive is, right? In some cases, if you've got a lead, you might have an incentive to have things regulated because it may slow down some of the startup or open source activities.

So, um, I'm not an expert at all of that, but I do know that it's here to stay. I think it's gonna fundamentally change the way that software is developed and, um, and delivered. And, and I think if we do it right, we can provide incremental, if not exponential improvements in the near term.

While also, you know, um, uh, kind of navigating towards what may or may not be, um, you know, agents taking over larger and larger portions of, um, of the software development lifecycle. What is that one thing you hear people talking about and maybe overly hyper focusing on that just makes you shake your head and go, folks, we're missing the point. Uh, it's a good question.

Um, look, I think some of the things you just highlighted there, right? I mean, I saw a demo this weekend of two agents going back and forth and building an application, which was super compelling, right? It was a, a really interesting, um, uh, use case and maybe a, a glimpse of what's, what's possible in the future.

But then I started to think about what would that mean to a large scale bank, right? What would that mean to a, to a, a, a, you know, a a healthcare company? And when you start to apply the realities of, of the regulatory environment, and when you try apply the realities of, hey, if this app is wrong, once, it may have much broader implications that is wildly different than, than, Hey, I, I, you know, I I created a, a, you know, a fake, you know, uh, app that, that is just using for demonstration purposes.

So that the thing that I, I think that, that, um, I shake my head a little bit is this, this notion that it's gonna just sort of magically be able to be, um, fully accessible and impactful, um, at the agentic layer in, in, in the enterprise. A again, I think we'll get there over time, and I think anytime you give a prediction without sort of an end date, like you're gonna be right. More often than not, you just don't know, uh, when it comes true.

ai is, is, is not allowed to be accessed on your or computer to embracing it, to using it as a, certainly as a productivity tool, helping to write more code, sort of code assist on steroids, if you will. And, um, and also as like a bit of a, a knowledge partner, right? To be able to, um, to ask a question, Hey, if I was gonna create this type of, of algorithm, what would a best practice be?

Uh, sort of the evolution of, of, um, uh, some of the online tools that we've all as developers been using for years, how we then get that into scale enterprise production, how it then, um, changes the way that we test and secure and other applications. I'm sure that, and we're working on stuff that, in that space, um, things like autonomous testing just point me to the app and the code base and we'll, we'll figure out, um, all of the, the test coverage that's, that's required, add in the telemetry usage data, and we can do even more, uh, interesting work. Those things are gonna happen.

But again, I keep thinking more and more that, um, the large scale changes end up happening more, uh, evolutionarily than revolutionary. And when we zoom out 10, 15 years from now, we'll say, boy, that was a big revolution. But when you're in the, in, in it every day, it feels like these incremental improvements.

There you go folks. I think we've seen this movie before, and one way I kind of described it as, uh, technology innovation is here. It's just being applied unevenly and AI is no exception.

Derek, thanks for being on the show. Awesome. Thanks Mike.

And thank you all for watching the latest episode of the Text Drawing AI video series. You can find this episode and others on our website. We'd invite you to check them all out.

Until then, we'll see you next time.