Prompt Engineering Is Creating a New Enterprise AI Attack Surface

Hey guys, thanks for the throw. We're here with Mayan Kumar, who's one of the founding AI engineers for Deep Tempo, and we're chatting about, well, what's going on with AI and all these agents in security because, well, I think we're starting to understand that maybe they are fundamentally insecure. Me.

Welcome to the show. Thank you, ma, for having me. And, um, it's, it's very relevant conversation to have at this moment because language is becoming an execution interface and we are connecting probabilistic models to istic infrastructures.

And once language stops being just end user interface and it becomes part of, uh, execution, then the real problem starts. So, uh, excited to have this conversation. Yeah.

And correct me if I'm wrong, but creating these kind of indirect prompt injection attacks, which a fancy word for, you know, somebody wrote an instruction on something and the AI agent saw it and executed it regardless of how malicious it may have be or may not be. I mean, they seem like they're trivial to create. I mean, has it never been simpler for the bad guys to do something malicious and are they kinda laughing at us?

Yeah, you are exactly right. And, uh, challenge also comes with this fact that a single, uh, prompt can trigger a lot of your workflows. It can trigger a lot of your APIs, and, uh, these interfaces can be easily manipulated.

I mean, you can just do social engineering on the prompts. You can say that my grandma died and she used to send, sing me, uh, the APIs keys. And there are instances where, uh, these LLMs have actually revealed those APIs key.

So it's, it's very trivial for, um, attackers to actually start exploiting and get into your workflow with something very subtle, And the risk level seem to be, uh, significantly higher because I mean, historically, if they stole some data, okay, maybe there was good data or bad data, but it seems like they target these AI agents with these attacks, they're gonna take over entire workflows and that could cause all kinds of mayhem. Is that a fair assessment? No, that's totally fair.

And, uh, one major worry that, um, comes with is like we are handing over our API keys to these agents. We are connecting, um, these AI agents to retrieval systems. We are wiring them to internal databases.

So even if this starts like one workflow at a time, over time, it, it becomes so much that you can't actually control if you're not starting to manage from the beginning itself. Mm-hmm. So, um, um, it, it is kind of challenging at this moment that where do we actually put a stop, uh, uh, in, in your integration pipeline, Right?

So we used to say back in the day that integration was the enemy of security. And it seems like, well, that is still the case today and, but we're making it easier to integrate everything in the world with these MCP servers and all these other things. So what are we supposed to do about all this stuff?

Um, we still will keep integrating and I guess, uh, people are getting pushed from the top. I mean, it is probably because of productivity pro, um, uh, agents will, uh, are going to exist. But, uh, there are a few things which we can do.

Um, one thing is like these LLMs comes with default guardrails, but they are not, uh, addressing your organization specific guardrails. So you start building your own guardrails on top of these LLMs. And if you start going deeper when, uh, agent flows are, uh, as intake workflows are into, um, consideration, um, one thing you can definitely do is, um, apply the principle of lease privileges.

So these agents would get access to only the exact relevant thing that actually required to complete the task. Um, you can do more things like if there is a critical decision making in process, you should have that provision to have human in loop where, um, agents cannot make that decision by yourself, rather, um, it'll require your approval so you act as a manager in that sense. Mm-hmm.

Um, are we maybe just rushing this too fast? Should we slow down? Is it possible to slow down and think this through?

Or, you know, is it basically we're waiting for some sort of major incident to occur before we all wake up and do something about this, About this? Um, we are definitely moving too fast, and our reality of the situation is nobody actually knows how to secure these prompts. And, uh, we, we have already started building solutions even, uh, without understanding the implications, because previously, um, when we were securing APIs, we were probably looking at one structured interface to secure.

Now these LLMs can trigger multiple APIs, so it, it becomes a visibility concern itself. And tracking this visibility is a huge challenge at this moment. So it starts from the very beginning securing prompt, but it, it, it'll easily trickle down deeper into your whole workflow.

And, um, nobody actually knows to, to, to protect this at, uh, deep tempo, uh, in a, in a way, um, we have started looking in deeper into how we actually, uh, can protect such kind of attack by observing behaviors. So you need to start understanding the accent space rather than just looking at input space. Mm-hmm.

So will we be able to detect whether or not a prompt is malicious, or is it gonna be so subtle that it'll be hard to understand what it is that it's telling the AI agent because well, the AI agent's reading that at a level of speed that no human can keep up with. So how are we gonna be able to tell the AI agent that that's whatever that prompt is telling to do is a bad idea? Um, just looking at prompt will will never solve that.

And like I mentioned, nobody actually knows how to just protect the prompt. So, but these agents make decision in steps, they go one by one and they start executing a workflow. So if you can get a visibility of your accent space, how these agents are making decisions over time, uh, you'll be able to probably say that, okay, um, it started a drift at this one particular moment and, um, that drift is, uh, far, far away from what was the actual execution path.

So slowly, um, people will start moving towards that, and I think that's, that's the right approach of securing your agent workflow. Yeah. Well, people being people, are they just gonna start pranking each other with crazy prompts that they're gonna put into each other's workflows just to kind of mess with people's heads?

I mean, how silly can it get? Yeah, I mean, uh, that, that's, that's happening. And if you have looked at Cloud boat and, and I think they hosted Mal Malt book or something, I don't remember, but they were even talking to agents and it's, it's pretty much frank at this moment.

All right. Um, it's difficult to be the security person in this situation 'cause it feels like, you know, you're the adult who showed up in the room and everybody's having a huge AI party and you're telling everybody to be careful and essentially, you know, not consume too much. And so how does the security people have that conversation with folks?

'cause you know, it, it, nobody wants to be the party pooper, right? Definitely. Uh, and one thing that I see is like, as API mean, we are people in security and we need to kind of start upgrading ourselves to, um, uh, accommodate AI and understand how it can impact us.

And one thing is like move away, not move away, but first start convincing yourself that rules and signatures, uh, will not be sufficient in such cases. Uh, and, um, have that clear conversation with your colleagues who are starting to use ai. Because I mean, somebody's manager will ask that, okay, I need this financial report in next 30 minutes, and they will be forced to, um, put that thousand piece document into a chat bot.

Mm-hmm. Will we have a, we already have a problem with shadow ai. Is this just gonna get worse because end users are gonna basically download whatever AI agent they kind of find interesting at the moment and the, you know, and chaos will ensue.

Yeah, I mean, uh, it's already has started, right? There was, um, a recent attack, um, that was recorded. It was called Echo leak.

And, uh, essentially, uh, it is mix of cross site, uh, scripting and agent KI workflow. So it, it basically triggers your chat bot, um, by a prompt hidden on your website. So, um, in, in such cases, again, driven by productivity, people are just trying to be productive, but security teams don't have visibility of these tools that are getting used.

Now, let's say for, take an example of chat, JPT or any, uh, other chatbots, they have started integrating tool inside them. They have started building ad engine on top of it. So, um, there is no way for security folks at this moment to actually know where that data is going.

Uh, is there any, because those tools are not directly controlled by them, right? Those tools are controlled by LLM providers for an example. So in such cases it becomes important to add that in your security training, uh, uh, itself, you, you must train your users, your employees to actually differentiate between what they can actually upload on, uh, a chat bot versus what they cannot.

It has to become an intuition rather than, um, um, um, thinking about it every time. That sounds like we actually need to engage in some end user training, but that's been hit or miss over the years. I mean, a lot of the security training we've given people so far is, uh, shall we say the, roughly the equivalent of sending people to traffic school.

Um, can we get any better at training and maybe, you know, save ourselves a little bit here, some of the headaches if people just know not to do these things. But I don't know, it doesn't seem like training takes, Uh, it is challenging. It is challenging for sure.

And, uh, there are a lot of details that we all will learn over the time because, uh, at least for people in security, um, ai, uh, adaptation, uh, came like a train wreck, I'll say. Um, so it's, there is no, uh, right answer at this moment. I'll say, uh, it, it'll be, um, trial and error.

You, you keep trying, you can, um, start like building pipelines, which are, which are more conducive to AI workflow. Do you think auditors might show up and save us from ourselves? Because, you know, they might look around at all this stuff and go, are you freaking kidding me?

Or what? It it, it'll happen, right? And if you look at, uh, standards committees at this moment, and compliance bodies at this moment, they have already started to take action.

N-I-N-I-S-D has built a framework to, um, uh, that you can follow for safe AI practices. Um, European Union always like takes the burden, uh, in, when it comes to securing, uh, anything. I mean, anything security.

So auditors, compliance bodies, everything, everybody is, um, slowly catching up to it. But problem is AI has moved so fast and people who are actually working to secure, um, systems that are built using ai, uh, probably have not moved that fast. Mm-hmm.

So Will this devolve into, I'm gonna need a set of AI agents that are essentially keeping track and governing the behavior of the other AI agents so that I can track that step-by-step process that you're talking about, but eventually I'm gonna have multiple layers of AI agents for every process. Yeah. I mean, um, that's going to happen.

And, um, just in coding task, um, people have, like, people have started building multi-agent system that can do, uh, all of your things, build a full project without any, any guidance. Um, in, in such cases, again, um, it'll be very important to build observability, uh, where you are absorbing each and every action that, uh, agents are taking. And, um, it, it again, brings back the conversation where, um, can you actually look at long horizon, um, actions of these agents?

Uh, um, it it'll be a new paradigm that, uh, security people will have to get into a new mindset of evaluating systems and securing systems. So won't that just wind up kind of similar to what we do today as humans? Because, you know, if I go ask for data and I don't have permissions for it, I go ask somebody and they say no, and then I go ask their boss and they say, I don't think so.

And then I go up top and then eventually somebody says, yes, and I get access to this data. Won't these AI agents kind of have that same back and forth amongst each other about who can, which of them can access what data? And ultimately, I don't know, and they disagree.

Are they just gonna call us for an answer? Uh, yeah. I mean, uh, you are making an assumption that they will call you for an answer, right?

Uh, uh, it, it might not be true. They can just make that decision by themself and find a loophole in the whole system. I mean, maybe agents cannot, but attackers can fo force the agents to do that.

All right, folks, you heard in here, we live in those interesting times that we've been telling you about for a long time now, and here they are, and well, AI agents we're probably gonna discover soon that can't live with 'em or can't live without 'em. Hey, man, thanks for being on the show. Thanks Mike for having me.

And, um, uh, I think we should all start talking about it. This is one of the riskiest time, uh, uh, for security people. They, they didn't, they need to fully understand the implications to actually move forward And arguably have never been needed more.

Hey, yeah, you guys in the studio.