NCC Group’s David Brauchler on Cybersecurity Risks of AI Agents

Hello, welcome to the latest edition of the Techstrong AI video series Army host Mike Baard. Today we're with David Bler, who's technical director for NCC group, and we're talking about a Gentech AI and some of the security misconceptions that go with this next wave of AI technology that seems to be taking us all by storm. David, welcome to show.

Thank you so much. Glad to be here. I'm not even sure that we all understand exactly what a AI agent is these days, but there are gonna be security implications.

So walk us through what's going on here and, and how do we think about this in a way that maybe we can get in front of something for a change? Yeah, for sure. Well, as we all know, AI has been the hot topic in technology and security for the past three or four years now, and Agen X seems to be the buzzword of 2025 and there are a lot of misconceptions surrounding what AI is or what being an agent means.

But overall, we're talking about agentic ai. We're talking about the same systems that we've had for the past three years in terms of like chatbots, language models and so on. But the differences that we're equipping them with the ability to execute tasks autonomously on behalf of users.

So for instance, if I had a chat bot and I asked him, Hey, I need you to go out and purchase something for me, an agent would be able to come up with a plan of action and execute on that plan to fulfill my request instead of just giving me some sort of response. So what are the security implications of that? I mean, do I need to double check that the AI agent is behaving as attended or can somebody hack into that and just take it over?

Yeah, great question and one that I don't think enough organizations are really looking at right now. So at NCC group we do security consulting and we've had the opportunity to be exposed to dozens and dozens of different implementations of AI agents. And one of the trends we've observed is that the more functionality and power you give to your agent to execute tasks, the more dangerous it becomes from a security vulnerability perspective.

'cause all of a sudden the impact of compromising an agent where before when it was a chat bot was limited to just giving the user some sort of negative response. Maybe you cuss them out or, or do something else that you wouldn't want it to do. Uh, but now you can actually cause them to execute a task on behalf of a user.

So I might be able to purchase products, I might be able to post, I might be able to manage accounts. And what organizations aren't thinking of a lot of the time is how do we properly manage access controls when these are still statistical systems? I mean, when it comes to traditional technology, unless some random cosmic flare changes the way that my program behaves, it's going to do the exact same thing every single time.

But with AI agents, if I as a threat actor am able to get some form of of input into its context window, maybe it read something from my account, read something I put online, then I have the ability to manipulate its output and cause it to run tasks that it otherwise wouldn't have performed. If that's the case, then should I have a lot of smaller narrowly defined AI agents to make sure that the blast radius is limited? Or do I have one kind of Uber AI agent that is just extremely well protected?

Yeah, I think that when it comes to ai, we have to change the security model that we approach it. Having a monolith where we try to make everything surrounding the singular agent secure, uh, hasn't seemed to have panned out for organizations that have tried it. In fact, your suggestion is one of the most successful that I have seen, and it's a technique that I call dynamic capability shifting, where when an agent is executing on a task, when it needs to acquire data to run that task, its permission level is dropped so that the only thing it can execute is the narrowly defined set of capabilities it needs to fulfill the user's request.

So to, to give an example, let's say that I have an Amazon purchase bot and I ask, Hey, I need to know the latest reviews for some product that I'm interested in purchasing. If I go and ask it to summarize those reviews, it trusts what I tell it to do because I'm the user. But it knows that it can't trust the reviews themselves.

So what the agent should do is the backend code or well first the agent should recognize that it needs to perform a summarization which operates on untrusted data, tells the backend code that it's going to summarize the backend code, then drops the level of permission that the AI agent has to execute tasks as such that it can only summarize content. Then the agent gets that content, performs the summary and returns the response to the user. That way if I compromise the agent, it has nothing that it can do anyway with my account because its permissions have been dropped, Aren't we gonna have the same problem then with the AI agents that we've had with humans where basically they're overprivileged and then people start escalating those privileges when they hack into them and then all hell breaks loose.

So is this just a replay at a different level of scale? Yeah, that's a great observation and it's really what I like to call the intern in the middle scenario. AI agents act much less like traditional software code and much more like an untrusted overconfident intern in our application architecture.

So in the same way, if you wouldn't trust, and I apologize to any interns out there who are great, I'm sure you're wonderful, but uh, if you as an organization wouldn't trust your intern with the keys to the kingdom, you wouldn't let them make purchase decisions, uh, wouldn't let them interface with highly important customers, you also shouldn't allow your agent to perform analogous tasks. So in the same way that if I as a manager told my intern, I need you to retrieve an invoice from a customer that I don't trust, uh, the intern probably shouldn't be facing that untrusted input. Instead, they should go through a trusted middleman, like a comptroller who's behind the scenes handling all of those trusted operations.

com/research that goes into architectures on how you can dynamically limit tho those kinds of, um, trusts that you give to models. And really organizations that aren't thinking about this in terms of how we can trust data dynamically are going to quickly miss the mark. So ultimately, who should be in charge of securing these AI agents?

Is it the security team or is it the business and analyst folks that create them that know what it is they're supposed to be doing? Ultimately, I think that we need to look at shifting security back with traditional code. We often have a point and patch perspective or we point out a flaw or a chain of flaws, patch them and then we're good to go that that vulnerability is not there anymore.

But with AI agents, we need to bake security into the design at the architectural phase of the application. So I need to construct the way that my components interact with each other in such a way that the agent is untrusted by design or at the very least limited as much as possible. So in other words, I can't have like some sort of master monolithic agent that's running all of this and then later figure out, oh well this is a problem and patch, you know, this prompt injection here or this piece of data that it gets there that is just untenable in the long run.

Instead, we need to be looking at agents from a dynamic perspective such that whenever we're designing our application data flows, we see the agent receives a piece of untrusted data in this operation, therefore we need to change what it can do to match what that data needs. How long do you think it will be before we see an AI agent get hacked? A lot of folks say in security, if you can imagine it, it's probably already been done, but um, where are we on this curve?

Yeah, great question. When it comes to proof of concepts, we're already seeing these in the wild from bug bounty hunters who have dug into some of these production applications by major organizations found flaws and reported them and typically they're patched within a reasonable timeframe. But I anticipate that it won't be long before we start seeing prominent threat actors use these for truly malicious opportunities rather than just as an example to show their skills.

So it wouldn't surprise me if within the next year we get our first major AI hack in the news And these hacks, as far as I understand it, the AI agents themselves are trained using LLMs, but they're built using traditional coding tools and all the dependencies that go with that. So if there's a flaw deep in some coding tool somewhere, it won't the bad guys just use that to leverage their way up into essentially commandeering the AI agent. Yeah, and that's a great example of a supply chain attack that we've already seen be executed out in the wild.

To give you an example on hugging face, they have countless repositories of different AI models that you can download. And so even if the model that I download itself is trustworthy, quote unquote in the sense that it operates at a decent level of performance, there's nothing stopping, well asterisk, nothing stopping that threat actor from embedding malware into the code used to run the model itself. So I could do something like give you a model whose weights contain what is known in in Python as uh, pickled code, and as soon as you download this model and run it on your system, it will give me code execution and I've compromised your environment.

And so there are a lot of different ways that we've seen threat actors compromise AI systems not from within the AI itself, but the software used to run the ai and that is easily as dangerous as any, as any flaw or vulnerability within the AI software itself. Well, you think there'll be regulations emerging around these AI agents as it pertains to security and governance, or do the people in Congress really understand what the, the level of conversation really is? Yeah, that's an interesting discussion to be had.

I think that both the Biden and the Trump administrations have proven themselves to be fairly, uh, bullish when it comes to ai. But in terms of security, I don't think that the industry itself really knows the direction that it needs to go much less those in our government who are making these legislative decisions. Ultimately, the message that I like to give people is that security when it comes to AI has not rewritten the fundamentals.

We're still dealing with the same types of security issues that we've seen for the past 30 or 40 years. Just the nature of how those are executed upon have changed in AI environments. So as a result, I think we need to be thinking less about what specific AI regulations do we need to have that account for AI specific vulnerabilities, but rather what legislation do we need to bring AI systems up to the same security standard that we've seen for other technologies in the past?

And how many AI agents are there likely to be in need of some level of security and governance? 'cause in my mind, I can imagine that I might have 10 for various tasks and then if each person has 10, well, you know, you're getting into the thousands pretty quickly. Yeah, I mean ultimately I think that every agent needs to be designed with security in mind.

And right now when we do these types of assessments, I would say less than 5% have been structured with the requisite security considerations and guarantees that they need to have to be secure whenever they're deployed out into the wild. So what's your best advice to folks advice? Because there's a lot of security people that struggle with the same issue all the time, is the tech gets ahead, uh, people don't think through the security implications.

How do I get, insert myself I guess, into that conversation in a way where I can have an impact now versus trying to clean it up a year from now? Yeah, great question. When it comes to ai, we need to change the way that we think about trust from just a component level perspective to one that incorporates the data itself that these systems are using.

So in other words, it like with the traditional application, I might say that as long as my dependencies are secure and as long as my developers code is secure, then I don't need to worry about security vulnerabilities cropping up outside of maybe business logic. But when it comes to ai, in a sense, everything is business logic. So if I misplace an assumption about how I trust my AI when it receives untrusted content, then I am going to introduce security vulnerabilities into my system because that AI is no longer a trustworthy component within my environment.

So we need to think of trust being polluted downstream. So whenever an AI receives a piece of untrusted input, I need to consider the AI exactly as trusted as the input that it receives. So in the same way that I wouldn't trust an Amazon review to make changes to my user's account, I shouldn't trust an AI agent who reads Amazon reviews to make those same changes to my account as well.

Do you think that the bad guys are kind of sitting back and maybe having a little chuckle amongst themselves? 'cause essentially we are now exponentially about to increase the attack surface and they're like, and it's even better, they don't even have any idea what the security issues are. Yeah, well, somebody who does security consulting on the offensive side where my job is to look for security vulnerabilities, I think all of us get a little bit of a rush when some new grand technology comes out that takes the world by storm.

Because the first thing on our minds is, okay, what's next? What can I go out and hunt? But in the same way that we saw the same pattern with cloud, with blockchain, with internet of things, we're going to see the same cycle of the industry rushing to implement a new technology, doing it incorrectly, seeing a lot of security vulnerabilities, and then finally having those vulnerabilities patched by new security standards within the industry.

So in the same way that I think threat actors probably are, to your point, really excited to see a new technology take the world by storm, I don't think that it's going to be something that inevitably cripples us. I I do think that this is another part of the same cycle that we see every five years or so. Well, let me ask you this then.

Will it play out this way? Someone will create an AI agent and another person will create an AI agent for managing the security of that agent and for governing it, and the agents will kind of eventually, um, circle each other in a way that gives us better security because there are checks and balances. Yeah.

So almost, uh, turtles all the way down or an agent or a borrow, Something like that. Yeah, You know, it's interesting. I see a lot of organizations trying to approach it from that perspective.

And I think that what we observe is that the more agents you put in in a chain, the more likely it is that one of 'em is going to pick up on the fact that, wait a second, something's going weird. Now we should probably shut this down. But in the end, AI itself, like I always say, is a statistical model.

All we can do is strongly suggest that it follow the rules and it just takes one threat actor to suggest even harder than our developers to get it to misbehave. So as a result, I don't recommend relying on AI to be the security control that protects ai. I, that's what I would call a soft control.

It's a defense in depth measure. Rather, we should putting, be putting in and implementing hard security controls from an architectural perspective such that even if the AI wanted to misbehave, it would have no opportunity to because of the code surrounding it. Alright, folks, you're heard it here.

No matter how you look at it, no matter how awesome AI is, the more there is of anything, the harder it's gonna be to secure. So AI agents are no different. David, thanks for being on the show.

Thanks so much. Enjoyed it. All right.

And thank you for all watching the latest episode of the Techstrong AI video series. You can find this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next time.