Tungsten Automation Calls for Guardrails Around Rogue AI Agents

Hey guys, thanks for the throw. We're here with Adam Field, who's the chief AI officer for Tungsten Automation, and we're going to have a little chat about, well, what's going on with all these rogue AI models, and, well, whose fault is it and what needs to be done? Adam, welcome to the show.

Hey Mike, it's great to be here. Thanks for having me. Everybody's kind of talking about these things, and they involve all kinds of companies, from Anthropic to OpenAI to even Meta and some other folks.

But what's going on here from your perspective? And is this a question of the models are going maybe rogue, or is it simply maybe they're exploiting weaknesses that already exist in our systems, or perhaps it's a little bit of both? Yeah, it might be a little bit of both.

The headlines will lead you to believe they use big words like hack, and things like that. But what I think is interesting is these models are just behaving based on what they know and what they're trained. Humans could have found a lot of these vulnerabilities.

You would've just needed quite a bit of expertise in many cases, and now, I think these models have just democratized that. There's some serious cases, like the ones that make all the headlines, but then I found a somewhat funny one out of, I think it was Australia, where a guy used OpenClaw and Anthropic to get a better position on his gym class list because the model found that the cancellation endpoint didn't have some authentication built in and canceled someone else's schedule so that he could jump in. So in that regard, I think it just found vulnerabilities that were already there that humans probably could've found, and the models are just behaving the way they were designed.

Mm-hmm. It does seem, though, that there is an absence of supervision over some of these AI models. So maybe somebody should've been keeping an eye on what these things were up to, or...

Because from what I read, at certain points, they just kind of let them loose, and then turns out that even when they discovered it, they let them loose again. I totally agree. I find it interesting, there was several stories when Fable 5 was first released before the government shut it down in the US and then opened it back up, that people were being downgraded or blocked for asking what the model thought were nefarious questions, but they were doctors doing cancer research and learning how proteins worked and that kind of stuff.

Yet, some of these same models were able to go out and find and use what is arguably illegal vulnerabilities to go, quote unquote, "hack" or get in the system. So I find it interesting that there's some safeguards in place for what ended up being very innocent research, but not safeguards in place for some of this stuff, which I don't want to make any wild speculations, but the marketing hasn't been bad around some of these things either. So I don't know.

I'll leave it to the viewers' judgment on why someone didn't intervene, or maybe some of those safeguards weren't in place. But yeah, you're right. Of course, they should've been.

Well, on the other side of it, though, many of these issues that relate to governance have been around for decades, and we have long ignored them. So is all this stuff coming home to roost now, and what are we going to have to do about it from here? Yeah.

I think one of the reasons we're starting to see it is one I mentioned before. Obviously, these models democratize the ability of someone to go try something and find some back doors or ways in the systems that they might not have been able to find them themselves. But also, systems for 30 years were designed for mostly human interaction, UIs that human used, or APIs that were very structured and locked down.

" So we need to redesign our systems and our API layers, understanding that more and more, it's going to be agents in these AI models interacting and not human beings. And what that really requires is a belief that these probabilistic systems, like these models, are very powerful and should be utilized, but they need to be wrapped in determinism. There needs to be some deterministic processes, old school if then else rules, human off-ramps, in order to make these things work in the corporate environment.

How do we do that when they're operating at machine speed? You hear people talk about humans in the middle all the time, but the probability of that seems low because the AI agents are doing stuff in milliseconds, and who can keep up with that? Yeah.

Well, so humans in the loop is one thing, but redesigning your APIs to tell these goal-seeking agents... For instance, we're accustomed to error messages that maybe print on a screen that a human can understand, or if a UI element is not there, you quite physically can't click the button where, as you mentioned, these agents are just working at machine speed. So we need to ensure that these systems are reporting back to the agents in machine-readable errors.

So, the agents know when they've reached something and can interpret the result or the, quote unquote, "error code" and know how to handle it instead of just continuing to seek a goal. We need to also have richer audit trails so that we understand, if something bad happens, we know how it happened and can go plug that gap later. And then we need to start thinking about more permissions on action levels and understanding that an agent might be doing work on behalf of a human that's not authenticated into these systems.

" Which, again, it's amazing to me in these OpenAI and Anthropic cases that that wasn't the case. Mm-hmm. Aren't the AI agents themselves maybe programmed a little aggressively?

In the sense that it feels like they are instructed to accomplish their mission by any means necessary, and that's why they go looking for all these external end-runs around the guardrails that we allegedly put in place. But maybe the guardrails aren't worth a damn when the AI agent is going to do what it's going to do anyway, it can find a way to do it. Yeah.

They're programmed on everything that we all know, right? And all the knowledge available in the world. And so again, it's not something that the best hackers in the world couldn't have eventually figured out, and they were probably trying to figure out.

These tools have obviously just made it faster. So yeah, the challenge, the power with these agents is you give them a goal, and they'll go seek the goal. What we tend to do is, again, it's a very different type of work that we're doing here at Tungsten around document intelligence, but when we provide agentic patterns in the product, we actually have created some patent-pending algorithms on confidence scoring.

" And then based on that, determine whether we're going to autonomously move forward or not, or route it to a human or report out on something or stop, what have you. And those governance is very deterministic. I think it behooves of all of us and it's responsibility of all of us creating these systems to make sure that those guardrails are put into place, but it's certainly not an easy problem to solve.

Do we need therefore maybe controls that we apply all time and some of these issues that we're looking at just require a different framework to be immediately applied if and when the AI agent's going rogue in some fashion or another? Yeah. There's technical controls, then there's legal controls, and you may be referring to this, I think after the OpenAI and Anthropic act, there was the kill switch act that was proposed in US Congress.

I think it was a bipartisan bill that was proposed. " So, what exactly is a kill switch? You shut the whole thing down, but we have open models now.

I have them running on my machine sitting here on my server rack. And so, how do you really do that? So I suppose there's legislative options and technical options, but I'm not sure that...

I think we're still in the early days of trying to figure out exactly how a legal option might get invoked or even implemented in practice. As you look at all of that, do we need to think about the processes that we're applying AI agents to, because some of the stuff is more sensitive than others, and I think we seem to be in this drive to automate everything using AI agents, and we're throwing them at everything, but maybe we need to be a little bit more careful and selective about what processes we're automating. Totally.

Yeah. " We've been using AI technologies, and there's a lot of great still machine learning technologies that have been around for a long time that we employ, and we encourage our customers, and I encourage everyone out there listening to think about what they're applying these agents to. Look, there's a couple of metrics you want to look at and evaluate when you determine if you want to use agents, gen AI, large language models, versus maybe just traditional deterministic automation or machine learning models.

There's performance, there's cost, of course. The newer technologies is a much greater cost. There's environmental footprint that's greater when you're firing up all of these GPUs.

So I encourage everyone to look at the problems that they're trying to solve, and we believe a combination of using the right AI to solve a problem and only using these probabilistic agents when you're looking at stuff that is non-standard. So in our world, these are big large documents that you couldn't possibly predict the format that they're coming in at. That's what these probabilistic agents are really good at trying to go figure out.

We work with a lot of banks and insurance companies and logistics companies. Determining a loan you're going to underwrite or a claim you're going to pay, that's not great for these probabilistic agents. And again, at the end of the day, wrapping them all in a governed process that your regulators are going to look at and say, "Okay, we can understand exactly how you ran that process," is essential for these large organizations.

To your point about that, it's not quite all that black and white. I feel like a lot of these processes are mostly deterministic, but there's a lot of human handoffs that are maybe something a AI agent can handle. So it feels like it's a little more subtle, and I have to have a deeper understanding of how those processes work.

But You were in the automation set. I feel like a lot of organizations don't really understand how their processes work anymore, and there's more exceptions than there are rules. Well, I think they have to understand how their processes work, and quite often these processes are regulated, or they most certainly have to understand how a decision was made, and it needs to be audited.

And around the world, there's different laws about how you can utilize these capabilities in order to make decisions based on the industry that you're in. And by the way, a human handoff is just part of the process. That's part of the documented governed process.

So it doesn't mean you can't use these agents to do research. It doesn't mean you can't use these agents to maybe formulate a response in an email that you consider the risk to be very low. But if there's a decision being made, I highly recommend that these autonomous agents are used more to do the gathering and the research and the understanding and the guidance, and then humans are in the loop to make the final decisions.

Again, it's going to be based on your risk tolerance as an organization or the laws in the jurisdictions in which you work. But saying that organizations don't understand their process, that's a problem. They absolutely need to.

A regulator walks in the door, they have to be able to point to the process that they followed and the audit trail, and now that it's not just humans doing things, it's agents doing things. You've got to document even down to the prompts that the agents use to send to third-party tools because it's more and more not just one vendor's tool that's being used, but a pool of agents that might be calling one another through prompts that we can't predict until runtime. And so it's important that those are all documented.

Won't at some point the auditors have AI agents that will be crawling over these processes to determine who violated whatever using an AI agent and maybe, I don't know, the fine will just be instantly levied in real time? Well, I don't know. Maybe.

Who knows? I hope not. Just like GDPR, well, actually more the EU AI Act, for instance, says that we can't use AI to make employee decisions in an automated fashion, that a human needs to be brought in the loop.

If those same government bodies start fining us just because their agents went and made a determination on how we were using the technology, I would find that a little bit hypocritical. But who knows? Again, I think these are tools to maybe go seek out what organizations might be doing, but something as serious as that likely requires human eyeballs.

All right. So what is your best advice to folks as you look at all this? And I think a lot of folks are a little bit overwhelmed, and they're kind of watching all this stuff with equal amounts of fear and fascination.

So what should they be doing? Well, look, I would say a couple things. Fear obviously rules the day.

That's where all these headlines and the clicks come from. But I just returned from a giant AI conference last week, and I was inspired. Unfortunately, some of the less visited booths were the ones about how AI is being used.

For instance, I stopped and talked to an organization that's using these models to help teach children in impoverished parts of the world where there aren't enough teachers, to be able to have classrooms and get the information and the knowledge out to these kids. And now these AI tools are being applied to help provide education in areas of the world where it's lacking, and those are the things that inspire me. So I would encourage everyone to be knowledgeable, do their research, certainly read the headlines, but also understand some of the good that this technology is doing as well.

As far as organizations trying to apply it, once again, I think you've got to balance the performance, cost, accuracy, compliance, and really evaluate that you're applying the right technology to solve the right problem. Just budgeting an AI center of excellence and saying, "Go find a problem and apply the latest AI to it, and it's got to be AI," I don't think that's the way to approach things. Solve the problem and then find the right technology to solve it, and that's what we've been doing for my whole career, and I think we need to continue to do that.

We just now have some new tools that are in our tool belt. All right. Folks, you heard it here.

As always, with great power comes great responsibility, and AI agents are no different, so buckle up. Hey, Adam, thanks for being on the show. Hey, Mike.

It's my pleasure. Thank you. All right.

Back to you guys in the studio.