Manoj Chaudhary on Emerging Cybersecurity Risks in the Age of AI Agents

Hello and welcome to the latest edition of the Techstrong AI Leadership Insights series. I'm your host, Mike Bazar. Today we're with Manoj Chad Ri, and he is the CTO for Chitter bit.

And we're having a little chat about AI agents accountability and guardrails and all that good stuff because well, we're building it, but I think once again, maybe we're over our skis a bit. Manoj, welcome to share. Thank you very much, Mike, and it's pleasure to be on your show.

All right. Just about everybody you talk to is building some type of AI agent. At the very least, they have a prototype, if not a few running in production.

But I kind of feel like we're all rushing to go do this without thinking, are we gonna put in place for guardrails and who's gonna be accountable for what? Because I think a lot of these AI agents are not just autonomous, but they're kind of, you know, if left unsupervised, they'll do all kinds of things we weren't planning on them doing. Yep.

So you're, you're, you're absolutely right. Right. So let's, let's look at how, how AI's being evolving, right?

Ai, as we know, as you just said, AI is evolving faster than the regulation and governance framework. You know, the architecture people implemented few months ago is now kind of getting outdated. It's, it's at that past, the AI innovation is happening.

With my personal experience, when we first wrote our first AI assistant, three months later, we rewrote the entire AI assistant. Why? Because the evolution in this space is much faster.

Just look at how many protocols have come in in reality by various different companies. We have MCP from Anthropic, we have agent to agent in from Google. We have a CP from, uh, which is, uh, agent communication protocol from IBM.

All of these has come in life in last six to nine months. So that's the pace at which it's changing. In fact, Google last week launched a P two, which is agent payment protocol.

If the things are evolving at that pace, what's happening is all the vendors and organization wants to have some kind of AI associated with them, whether it is AI agent, whether it is AI assistant, whether it is agent ai. And in that hype, the security and governance is left behind. If you look from the governance and security regulatory standpoint, there is only one ISO standard available for ai, which is ISO 40 2001.

And that also has launched in last six months only. And what it promises, it, it comes and makes sure the companies are designing, developing and deploying AI technology with the transparency, data quality, security ethics, trust is what it's doing. So that's what is happening, essentially.

And I feel like there's two conversations that are closely related, but one is security and the other is governance. When I think about the security conversation, it almost seems like maybe we don't understand the real level of risk that's going on here because these agents are autonomous and if they get compromised, won't the bad guys just take over an entire workflow rather than just kind of compromising a a particular endpoint? Yes, you, you, you're right.

Like the, the, when AI agent comes into play, let's talk about that and I'll, I'll talk about MCP from same angle, right? So if you look, as you said, AI agent is autonomous, if the security is not in play, there can be so many things which can go wrong. Not only bias, not only basically prompt injection, people can inject prompts into the AI agents and cause it to do something really bad for that business.

And guess what? Not many companies are putting human in loop to verify. In fact, I believe that verifying the output of an AI agent is going to become in itself a profession.

So people are even right now with current, uh, innovation and early stages of ai, in my opinion, human in loop is a very important aspect. And, and many AI agent is basically taking human out of the question equation, right? So these security things needs to be implemented.

If you look at MCP, in fact, I put, I posted a LinkedIn, um, uh, last week because many people asked this question to me, what about security when it comes to these new protocol, you know, MCP model context protocol, which is delivered by an anthropic, it is a protocol to build, make the development of AI agent easier, but the security is ignored in this. There is no clear standard for authentication. There is no clear standard for sandboxing the AI agents.

There is no clear standards for making sure that tools are secure and connecting to the database, the data sources underneath it securely. And there is no clear guidelines on how to protect the prompt injection, for example. And then on the other side of it, with the governance, it seems like a lot of these AI agents are leveraging various large language models, all of which seem to be very, um, aggressive about hunting down data sources.

And if I don't restrict them, they'll incorporate that data whether I intended to or not. And won't that just one day result in, I don't know, some embarrassing data showing up in an output in some way that we didn't plan for? Yeah, so look, I, I think that that's the key thing that mo the moment the data leave and go into LLM, you are at risk of that data getting exposed to the author world, period.

That that's how I see the world. And that's how the reality is that that's where if you look at the short-term memory, long-term memory is in place for these AI agent. If you look at one of the AI assistant, which is we all use and which basically is all this gen AI evolution happened from them is Chad, GPT by default, anything you send to chat GPT is used for training their model.

You have to go explicitly uncheck that setting and still we don't know what they're gonna do, who, uh, uh, not many people go read all the fine prints. So what we need to do and what organization has to do is when they build AI agent, they should make sure that AI accountability is in place. What does AI accountability means?

Any input going into LLM, any output coming from LLM? Any input going from LLM to AI agent to invoke tools and make decision needs to be checked against the guardrails and make sure that they are valid and doesn't gonna cause the leakage of the data doesn't cause any compromise of the data sources. To your earlier point, I don't think that a lot of folks are actually reading any of these end user license agreements any more now than they did then.

And but today you have more at risk because somewhere in there you're supposed to opt out and check a box that says, I don't want my data to be used to train the AI model. But you gotta go find that box, right? Yes, absolutely.

And those bots by design are hidden. And if you look at most of these LLM providers or a, or where you are giving your data by default, use your data, your prom, your document, your data sources to accumulate and train the future models, which is gonna come. 'cause that's how the models are becoming intelligent.

That's how models are becoming smart and not only about becoming intelligent and smart, that's how they can give you a personalized experience on the task you are asking these models to do for you, right? That's basically is why they need this data and that's why they're basically having these boxes and opting out hidden from the users To that end, who's in charge of all this? 'cause I think that we see a lot of data science tiger teams that people have spun up and they're out creating agents, but it's not clear to me that anybody from security or the GRC team is invited to that conversation.

So, uh, are we just waiting for some sort of cataclysmic event before we get serious about this or what's gonna happen? You know, there are a bunch of events already happen. A as we all, as majority of people might know, or your audience, Samsung IP got compromised because one of the employees of Samsung literally take some confidential document and put it into the chat GPT, right?

And that is now, and that is available on the internet, right? So this is happening again. The point is the people want, the organizations don't want to left behind and the innovation is move on, on AI is moving at much faster pace, whereas the security team and governance is lagging behind and not able to catch up with the innovation on that development of ai.

So yes, I think there will be going to be some catastrophic events before people are gonna get serious. Although I would say there is a positive news where majority of analysts and majority of big enterprises are now asking the diff various organizations who are building AI agent that, Hey, please take AI accountability. Please take AI guardrail, please take AI security into account.

Now how much is gonna happen in next six months? It's still to be seen, but I'm happy to see that now people are talking about these, whereas six months ago nobody was even discussing about the guardrails and accountability. So what's your best advice to folks then about how to go and get into the middle of this conversation?

'cause I think a lot of it seems to be happening beyond the realm of the governance and security people. So do I gotta go scan for these projects and insert myself into them or how do I kind of get myself into this conversation? Yeah, I, I think basically there are one few open source tools now built.

You know, there are vendors like us who basically are building AI accountability and governance into the product. Get yourself educated with that. If the cycles permanent, go look at the control of ISO 40 2001, which allow users to get educated with how to design, develop, and deploy, um, AI ethically and not have problems.

That's what I would suggest people to do. And make AI security and governance not as a second thought for ai, make security and governance as a first class citizen while building an AI agent. Build your team and train your team around AI controls.

When I say your team, your design team as well as your development team, along with the security on the controls you need in ai, Might we one day have AI agents that are gonna manage the security and the governance of other AI agents? Is that where we're headed? Absolutely.

You know, what's gonna happen is AI agents are like digital workforce like US humans. There are people who are developing and there are people who are doing security and governance on the things developed by the development team in the AI agent world. There will be AI agents which are responsible for doing the work, the task autonomously, and there will be AI agent, which are watching them for the security and governance and making sure that they are doing the things ethically.

There is no breach, there is no violation of anything. Absolutely. That's what is the future and that's where we are gonna go.

We've also been struggling to manage both human and non-human identities for a long time. Now, is an AI agent essentially a new type of non-human identity or is it an extension of our human identities and we'll track it that way? So right now it is where we are and at what stage we are, I would say it's an extension of human identity, but pretty soon it'll become a standalone identity of itself, which as I said, will be a digital workforce.

There will be, uh, agents which are basically onboarding these AI agents and there will be like, call them the HR agents, which will be responsible for onboarding the agent into the IT system. There will be AI agent, as we discussed, to make sure they are operating and performing securely according to the policies of the company. So in, uh, pretty soon not in a distant future, they will be the digital entity of their own.

Are you at all worried that we might get too comfortable with these AI agents and not think all this through and we're just gonna have people kinda executing things just because Well, the AI said it was okay. I I, I honestly don't think so because you know, a lot of, there is still a lot of hype around AI agent. If you look at the companies who are building true AI agents are very handful.

Lot of companies are building AI chart work and AI assisted, which are basically sitting next to humans and basically helping humans to do the task at much higher productivity level. There are very few companies are building AI agent, which are doing, making decision autonomously and operating autonomously. That's basically our very few companies.

And once, by the time this become a commodity, the true AI agent become commodity, I hope and I think the standards will get caught up and it will basically have governance around it. Do you think the auditors out there are already tracking this and will soon come knocking and asking people about these issues? Absolutely.

Like that's where ISO 40 2001 is come. I think there are more standard and compliance, which is happening. It's just matter of time.

It's these, these are regulatory authorities will come and start putting them in place. All right. Well folks, you heard it here.

I guess there's two things to remember about AI agents. I mean, they're awesome, they're powerful, but they won't pay the fines for you and they certainly won't do any jail time. So be careful.

Mano, thanks for being on the show. Thank you very much, very much, Mike. It's pleasure.

All right, and thank you all for watching the latest episode of the Techstrong AI Leadership Inside series. You can find this episode and others on our website. We invite you to check those all out.

Until then, we'll see you next time.