Securing the Agentic Frontier: Why AI Automation Needs a Human Handbrake

Hey guys. Thanks for the throw. We're here with Dr.

Aki Rashid, who is AI lead for Glass Wall. And we're talking a little bit about, well, we happens when we apply AI to security operations. 'cause sometimes things don't turn out the way we intended.

Akeem, welcome to show. Thanks for having me, Mike. Uh, lovely PM board.

So walk us through, what's the problem here? I mean, honestly, we've been trying to automate security operations for as long as I can remember, and it's always been with some mixed success and some failures. And now we're throwing AI into the equation and well, what can go wrong?

But the question is, um, what are you seeing out there? What are the challenges? Yeah, sure.

Um, there, there's a number of challenges here, Mike. Uh, I, I think what AI has done is, has just simply exploded those challenges. The, the scale has just shot up.

It, it has increased, uh, albeit exponentially. Um, you know, but the, the challenges have always been the same. Uh, with this, with this problem, you know, what we are seeing is companies, organizations, enterprises, governments, making use of ai, but then they're not quite too sure in terms of the, the more common things around, uh, security principles and trustworthiness of ai.

And, and especially when you get into the agentic AI realm, um, where you have these very complex workflows at times, and they can be quite unpredictable at times in their behavior. And all of these components interacting together can often lead to some, some undesirable results. And we've seen quite a few of those, uh, recently and quite some, some large organizations as, as well.

And for example, uh, we can take the, the recent or semi recent incident, uh, CloudFlare back in, uh, November, I believe it was, where they had, um, you know, they had, they had an ML based system, uh, which was listening for some kind of component file or something like that. And because of a, a misaligned input or, or an input that was undesirable for the system, um, that update propagated through the ML based system. And it had devastating consequences for the internet at large.

You saw large websites go offline. You saw, you know, banks getting affected. You saw, you know, flights being affected as well because of this problem.

And this was simply because of, you know, unvalidated inputs making it through into a production system when, you know, you could have had better guard rails in place. So that's just one small example, um, that I can, I can think of at the top of my head. Mm-hmm.

It seems like the challenge too with AI is that a lot of the things that we're dealing with, um, in terms of the output from a gen AI specifically mm-hmm. Are probabilistic. And I have a lot of workflows that are, shall we say, determinist dig in the sense that they're supposed to be done the same way every time.

And, well, Jenny AI doesn't do the same thing the same way twice. So how do I kind of meld these two things together? Yeah, that's, that's a, that's a very interesting point there.

I mean, uh, with gene ai, the inherent nature of it is that, you know, it's, it's trying to be creative, it's trying to be generative. It's trying to, um, you know, you know, expand your horizons and your knowledge as well with its generative and, and non-deterministic, um, capabilities. And I, I think it all just comes down to validation, and it all comes down to, um, good specification and also it comes down to good prompting.

So the entire end-to-end life cycle of deploying a JI system or, or, or a JI workflow for a particular problem, you know, is pretty much the same. It's the same recipe as any other software engineering project or any other high stakes mission critical project where you would want to plan things as well as possible. But then in terms of actually, uh, deploying such a system, you'd want to ensure that you have those safeguards in place for ensuring it doesn't go rogue, so to speak.

But, and equally, you should be able to, um, assess how it's going to respond, uh, with particular types of inputs way ahead of time before any kind of system like this reaches production. You'd want to do rigorous testing against this. The beauty of a probabilistic system, of course, is its predictive capability.

It can, you know, for example, in, in the, the, the concept of malware detection, for example, it might let you know about completely unknown and unseen threats that have not been captured in the wild at all. And it might give you a, an 80% probability or 90% probability of that happening. Um, but of course, that doesn't give you that, that kind of certainty, which, you know, we, we so often desire.

So I think there's a, I think with, with AI and these agent workflows, we are, um, we are in, in dire need of just following the existing protocols that are exist when you're trying to work with very high stakes mission critical systems where we need that kind of validation, robustness and, and those kinds of checks in place. Of course, a wise man once said, you know, it's one thing to be wrong. It's another thing to be wrong at scale.

And that's kind of the issue with ai. But, um, what's your advice to folks about how to go about doing all this? I mean, 'cause I think everybody's interested, but a lot of times I talk to people and they're like, well, the best people we have are the ones to do this, but they're too busy fighting fires to actually go automate anything.

So, you know, how do we kinda get to where we need to be? Yeah. So I, I think it's partially, um, an education thing.

This is a, a new technology for a lot of people. Um, I, I would, I would certainly go down the route of educating users around the, the, the high level risks of this technology. So whether it be, uh, an an LLM that you're interacting with or, or a chat bot or maybe it's, uh, an agent workflow, we've seen a lot of, um, recent developments in the area as well, and understanding what the, the common risks are there.

So prompt injection attacks, uh, data poisoning, um, understanding that you can't always trust the output. Um, and that also speaks to, you know, the, the issue of prob probabilistic outputs versus, uh, non-deterministic outputs versus deterministic outputs. But then in terms of, uh, your, your, your so-called, um, experts who, who are, who are fighting those fires, I think it's, it's on them as well to, to champion safe AI and trustworthy AI and promote, um, the, the safe use of, of these technologies.

So not only does that extend to educating the, the less, um, AI savvy folk, but it also means that they should be working rigorously and, and tirelessly for, you know, ensuring that new kinds of models, new kinds of protocols, any kind of new technology stemming from this is used in safe and in trustworthy ways. I wouldn't, I wouldn't necessarily say in regulated ways, a regulation can sometimes, uh, uh, come across negatively, but I think where there is common consensus among the AI community about what is right and what is wrong, um, that in itself is, is a form of regulation. It's almost like self-regulation on that point in time.

Mm-hmm. Um, will it get easier to automate things? Because I think part of the problem historically is we had soar, which was a great idea, but I had to have a lot of programming expertise to kind of make that work.

And if I have in a more of a natural language interface, can I, can I democratize the automation a little bit more? And just let you know, mere mortals kinda automate some things. A hundred percent.

Mike. Um, you know, the, I think, I think a lot of people who haven't, who haven't delved too deeply in the, the, uh, agentic or the, the recent developments around, uh, uh, agentic workflow, agentic AI and workflows and, uh, recent developments like open claw and Claude code and, and stuff like that will be, be, dare I say, pleasantly surprised. Uh, some people might be, uh, unpleasantly surprised at the, at the capabilities that that technology now brings, where you can give it a task, uh, which might have taken a, a team of humans six weeks, five weeks, four weeks, whatever, but then it can knock it out in a matter of hours.

Um, you know, and when you can run those kinds of, uh, tasks or that kind of, um, way of working in parallel, so you've got multiple agents or, or multiple ais working on several different tasks, whether they are distinct or whether they are interrelated, um, you know, that is a, a complete game changer here. So I think the simple answer is yes, uh, even for, for the, the the lay person and the AI space, I think things are going to change quite considerably if they haven't already. How do you think that will play out?

Because I can imagine a world where everybody on the security team has their own AI agents, and then the organization probably has some AI agents that are assigned particular tasks, and they're a little more autonomous, shall we say, but, um, won't these things have to kind of negotiate with each other, and who knows, maybe they'll even argue with each other, and how do I kind of figure out what to, uh, and supervise here? Exactly. Because they won't Yes.

All call back to home, which is us and ask for resolution. Yes. Yeah, exactly.

So that, that's a very interesting point there, I think. Um, so there's, there's a couple of things to unpack here. So of course, with this, um, extended automation or this, this increase in autonomy comes the, um, you know, the, the increase in risk that is associated with it.

So the risk of things going wrong, the risk of conflicts as you're describing there. Um, additionally, the risk of moving too quickly and generating a lot of AI slop, uh, which is a, a term that has been coined recently to describe, uh, outputs of ai, which aren't very useful. But, you know, they, they appear to be, uh, somewhat useful in the short term, but long term, they're just quite, you know, they're, they're very difficult to maintain.

It could be software, it could be, um, it could be copywriting, which doesn't quite make sense or wherever it is. But when you're using AI like this in this kind of agent fashion, you will of course get those, those conflicts where they, they are gonna, uh, require the human in the loop to make certain decisions. So I think the, the important here thing here is to, is to understand that these agents aren't, um, completely autonomous and they just shouldn't be used in that way.

They always require that human supervision. Um, and when they do inevitably call to home, you have to understand what am I trying to achieve here? And if the AI agent or, uh, the these, uh, these ais are setting out to do something which you're not trying to achieve, then I think that's a, a good place to pull up your hand break and, and, and, and stop what they're trying to do.

To your point about that, um, it seems like, are we involved in some sort of AI arms race? Because everything you just described, I think the bad guys are probably doing the same thing with ai, right? Oh, yeah.

So, and yeah. Is this all starting to be a battle that's being fought at such a level of speed that it's not really possible for humans to take care of it on their own without some help from ai? We're absolutely in an AI arms race.

Um, I, I think it's, I think it's a very interesting period that we're in. Uh, people are using AI for various reasons, some good, some bad, some extremely bad. Unfortunately.

Um, different nations are, are also using ai, uh, against each other. Um, uh, you know, you've got companies like Anthropic who've recently published some information around this topic as well. Um, when you apply AI to certain domains like cybersecurity, which are inherently, uh, adversarial, you always have those cat and mouse games in cybersecurity, um, you will find that the good and bad actors, so where you have people using this technology for, um, you know, for for defense, where you have people using this technology for the common good, you'll also have people trying to create malware with this technology.

You'll have people trying to create spam emails or, or very, um, sophisticated, uh, you social engineering attacks with ai. Um, unfortunately, this is just a, a, a trend that we see of any kind of new technology. com bubble.

Um, you know, when the internet, you know, it has good things, it has bad things, and it's a similar pattern now that we are seeing with ai. The thing with AI, of course, is that the scale has, has completely changed as well. You can now do things, um, at, at a far greater level than you could before.

So the, the, the so-called blast radius is almost unchecked. Um, and therefore you need people in positions of, of power and authority who know what they're talking about, who know, uh, what to do in certain situations, but also to, to build things in a responsible and safe manner as well. Mm-hmm.

And the one thing that also comes to mind here is like, let's say I'm the security team and I have built and deployed a bunch of AI agents, don't they in themselves become a primary attack vector? Because the bad guys are gonna be like, Hey, if I can get after those, well, that's the keys to the kingdom, right? Absolutely.

Absolutely. And, uh, that, that is a, a, a, a trend that we are now seeing. We're seeing entire companies being spun up and getting a lot of, uh, VC capital, um, you know, because of this problem of how do you go about securing your AI agents and your AI workflows because they just become the, the attack factor at that point.

And especially when those, um, agents aren't necessarily calling to home, but are calling to other sources, or, uh, they're making calls on, uh, to, to the internet. Maybe they're pulling packages from GitHub or, uh, whatever it might be, and those sources become a little untrustworthy or compromised. That is a, an attack factor that we've seen pick up quite recently as well.

But certainly we'll see in the future as, uh, AI agents become more interwoven in the, uh, in, in the enterprise landscape, in the organizational landscape, then the attack factor will shift. Of course, it will shift from spam emails and injecting things into, into, um, in, into spaces which humans tend to, um, uh, delve in and occupy and read into stuff that AI agents will occupy and read and consume, consumes that. It could be, it could be skills MD files, it could be, it could be, uh, plugins for AI agents, it could be software packages, it could be a whole host of things.

Mm-hmm. Um, so as you kinda look at this, are we deploying AI agents faster than the security teams can keep pace with? And are we just kinda waiting for some sort of catastrophic event before everybody goes and addresses all these security issues that are gonna require a lot more automation than we have in place today?

Well, I hope we don't, uh, I hope we're not waiting for that. Um, uh, and certainly, um, you know, the people I speak to and the organization, uh, that, that I work for, we, we we're trying to get ahead of this problem. And like I said, there are entire companies now set up to, um, deal with this problem of securing agent AI and securing AI and the, the ramifications of bad and untrustworthy ai.

Um, I think it's, the problem will get more pronounced because as the frontier AI labs and frontier AI companies, uh, inevitably release better models, inevitably release more features which make AI more accessible to people, uh, what we'll see, of course, is that the risk will exponentially increase, and therefore the, the market size for solutions to solve this problem will also increase as well. And we'll see, we'll see problems and risks that we probably haven't thought of before, um, simply because of the pace of change in this field. But, uh, my, my feeling, my strong feeling is that, that the community is working proactively to trying to, to trying to remediate some of the, the known, the known issues.

Of course, of course, there are always going to be issues which are, uh, kept by the adversaries, uh, as, as, uh, highly guarded secrets so that they can, uh, exploit at the right time, but about what we know around the common vulnerabilities that exist and common attack patterns that exist work is being done to try and address those as much as possible. So right now, what's your best advice to your fellow cybersecurity folks out there about, you know, how do I go have this conversation? Because sometimes they feel like, you know, they're the people at the AI party telling everybody maybe not to drink so much punch, if you know what I mean?

I, Yeah. So I, I think, I think it goes back to that, that original thing that I said, uh, at the beginning, uh, of our discussion, Mike. I think it's about trying to, um, en ensure that you're not letting any kind of AI system or, or system, uh, run rogue on any kind of infrastructure.

It's about ensuring you have those deterministic structural controls in place, just like you would have for any kind of IT system or software system or any kind of employee working at your organization, where you'd have proper controls in place to ensure that, you know, you only give access to information or tools or, uh, ip, which they absolutely need to have to, to function in their role. So it's a similar thing, uh, with AI as well. And where you are trying to deploy AI in a high consequence or, uh, high stakes or mission critical environment, increased oversight, uh, is going to be absolutely essential.

Essentially, it's, it's, it's, it, it can't be the fact that you let this, this kind of technology run itself that is just not going to to work. And you know what, we'll, what we'll see, of course, is cybersecurity professionals, especially, um, becoming more accustomed to securing AI agents and securing agent workflows and trying to, you know, try to come to terms with the fact that not only do they have to deal with their, uh, you know, with the humans who might raise support tickets and might, uh, raise issues or, uh, attackers who are humans, but they'll soon have to deal with, uh, agents who might, uh, require certain permissions to do certain things to achieve certain goals, but also agents who will try to infiltrate systems and attack systems and break them as well. And I think that a, a large part of this is that education that I, that I hinted at earlier as well.

Mm-hmm. Well, folks, you heard it here. Hey, the good news is it's gonna be a lot easier to automate things using ai.

The bad news is that more things than ever are gonna be highly integrated because of ai. So when something goes wrong, it's gonna go really wrong. Hey, ake, thanks for being on the show.

Thank you, Mike. Thank you. Cheers.

All right. And back to you guys in the studio.