Generative AI Threats with Perception Point’s Tal Zamir

This is Textron tv. Hey guys, thanks for the throw. We're here with Tal Zamir, who is CTO for Perception Point, and we're talking about all these new AI tools that people like Apple and even Microsoft they're creating and they kind of creates some level of cybersecurity threat that maybe we're not all aware of.

Tal, welcome to the show. Hey, Mike, thanks for having me. Uh, great to be here.

I think specifically the one that we're gonna talk about is that Apple is applied AI to email, and that is a, creating some interesting challenges because I guess, um, anybody who's on that thread suddenly can use that information for all kinds of malicious intent. So what should we be worried about? Definitely.

You got it. So just to give some background, um, recently, uh, there was a big Apple keynote where they unveiled Apple intelligence. Basically a bunch of AI features that will, will be built into iPhones, both for consumers and for businesses.

Uh, it's not rolled out yet, but um, probably by the end of the year we'll see this in action in real iPhones and in enterprise environments. And one of the features that caught my eye there security wise is the automatic summarization, uh, of emails that, um, users will receive. So basically when you get an email, maybe it's an email thread with a lot of previous, uh, conversation.

Uh, apple wants to help you, uh, prioritize and get the gist of it. So that'll give you a quick, uh, summary on the top of the email, uh, generated by, uh, generative AI large language models. Um, they also, uh, added, uh, the same kind of capability on your notifications on your iPhone, so you can get like the summary of what has been going on, on email, on WhatsApp messages and other types of applications.

Um, and one of the risks that I don't think, uh, many have, uh, focused on, uh, with this technology is how attackers can abuse this and pr practically confuse users and get, uh, their phishing and other types of social engineering attacks to more successfully, um, penetrate an organization. Uh, they're already using, you know, a lot of social engineering and sophisticated email and evasion tech tactics to avoid being detected by users, convincing them that they're real. Uh, but now with this automatic summarization technology and generative ai, uh, a smart attacker can compose an email as a reply to an existing thread or maybe, uh, even just a new email to a user where because of the summarization, they can hide the real intent of the email and get the user to ignore some of the fine print.

Uh, and by doing so, uh, they can be more successful in getting people to click links. Um, one example that pops into, and maybe you have a question, uh, let me pause here for a second. Well, they can also use that information to kind of craft a more sophisticated, say, business email compromise where they're trying to get into the workflow of the organization and understand what's happening.

Is that about the gist? Yeah, I think definitely, uh, of that example you can think of. Um, for example, let's say you have a conversation with your vendor, right?

Um, and on that conversation, many people might be CC'd, um, you know, some finance people, some, uh, of your clients. And if one of those accounts was breached by an attacker, the attacker can now send an email that looks legitimate, comes from a real, um, email address of one of those, uh, recipients. But once they can introduce new text into the tread, um, and we know that the AI will look at all of the text in the conversation and will try to summarize it, uh, attackers can build sophisticated prompts or, um, jailbreaks as they're called these days, uh, to try to confuse the AI into summarizing, uh, in a different way.

So they can create a summary that ignores the, the real intent or gets the user to do actions that they didn't intend to. Um, so they will just reply to this thread with some malicious text that will confuse the AI and get the summary to present something else. Mm-Hmm.

What are we gonna do to defend against all this? 'cause you know, apple especially was touting how safe their approach is 'cause it's all within their network. But, um, it seems to me all you gotta do is get the credentials and off you go.

Yeah. So this will be a challenge for sure. Um, I think, well, apple definitely takes privacy and security seriously.

Um, but, uh, apple, like many other vendors are now in this gold rush to deliver AI features quickly. Everybody wants to be like, uh, the innovator and sometimes I believe security might be left behind. Um, so I think it'll take them a few iterations to figure out all of the gaps in this system and make sure they have the safety controls in what the generative AI can do on, uh, people's email.

Um, and even if they don't send any information back to Apple Systems and it's all staying on your iPhone, even the fact that this AI, um, changes the perception and the inter interpretation of the email by the users, this can already have some serious impact. So they will need to put the safety controls and, um, what that AI is willing to do when getting texts from all kinds of, um, malicious actors. Uh, and I'm sure they'll nail it eventually, but we need to be very careful, uh, when it's first introduced by, um, looking at the fine details on emails, educating users, uh, to read the fine print, to understand what they're clicking on and so on.

Is this gonna be a problem across all our applications because they're all basically using Gen AI and creating these summarizations and, um, are, are we all thinking through the security implications? Definitely. You're spot on.

I think, you know, we already saw that Apple as another example here. It's not just introducing, um, AI into iPhones. It's all over the Macs operating system, across all applications everywhere.

And we'll see more and more of that. We have chat GPT now, uh, as an application on your Mac that can look at any data, anything you share on your screen and give you, uh, gen AI capabilities. And we've seen that even, um, you know, serious companies like OpenAI that have billions of dollars, uh, and surely have serious security teams, they have, um, security flaws.

Like one example is, um, when they introduced chat GPT like a year ago, a user that signed into chat GPT could see the titles of conversations of other users, um, which, uh, it doesn't have any authority to look at those conversations. And still it was exposed because of the, a design flow, uh, by OpenAI. So if it happens to OpenAI, it can definitely happen to the gazillion other, uh, vendors that are now baking quickly generative AI capabilities into their systems.

So definitely reason to worry. Um, and being, uh, extra careful with introduction of gen AI technologies in the enterprise. Aren't we gonna need AI tools to secure our AI tools?

I mean, how will that all play out? Yeah, uh, definitely. And it's already happening.

Um, us at the perception point. Uh, we actually already also of course, uh, use generative AI in large language models to protect our customers in, uh, email, in browsers, in cloud collaboration apps and many other areas. Um, for example, uh, we now have the capability, uh, to understand whether a certain email is malicious by looking at the intent or the semantic meaning of an email, uh, upping our game from the text level, uh, which were, uh, the traditional security solutions would look for signatures or patterns in the text.

And now we're moving up to the next level of looking at the meaning or the semantic level of an email or a website or whatever you might bump into as a user so that we can better understand whether the intent in that email or that website that you're visiting is malicious. So we are using a lot of generative ai. This is just one example, uh, to protect against, um, AI based attacks.

Who's ultimately gonna be in charge of sorting all this out? 'cause it sounds like, it sounds essentially in our workflows, and I know the security people are gonna be involved, but how do we get the average business executives to kind of wrap their heads around this? Yeah, I think, um, it is a priority for many CISOs to, uh, make sure that development processes, if they're a tech company or a company produces products to make sure that they bake in proper generative AI security controls, uh, when they're building gen AI products, but also companies that consume gen AI products.

They need to, uh, definitely, uh, adopt all kinds of security solutions that take this, um, as a major consideration. Um, and, um, I think the CISOs, um, are well aware of this and I, I hope that they will, uh, prioritize this and make sure they have the proper controls. Um, and this is like multi-dimensional thing.

You have protecting your own apps that you're building with generative ai. That's one, uh, you need to make sure your employees know the risk of using generative ai and they don't give out sensitive information to generative ai. Uh, and you need to protect against, um, attackers leveraging generative ai, uh, in their advanced attack techniques.

And I think, um, from what I'm seeing, there's a lot of interest, uh, with CISOs in, uh, making sure they're covered in these fronts. Alright. Do you think the vendors who are providing these services are aware of these issues or, and are they gonna advise their end users or somewhere buried deep in the end user agreement?

I'm sure. Yeah, definitely. Definitely.

I, I think vendors, um, are starting to be more and more aware of this, uh, when they introduce gen ai. Um, there's a discussion on safety and ethics in using gen ai, um, and hopefully regulation at some point, uh, on what is allowed and what, uh, uh, what kind of data, um, we should be able to provide to Gen AI and, uh, which control should be in place before we provide it. Um, you know, and this is a, will be a challenge together with the fact that everybody is trying to rush into delivering those features quickly and to show, uh, how we are the coolest, uh, um, vendor with the best gen AI features.

So there's a trade off and hopefully companies will make the right choice and put the appropriate controls in place. What's your sense of how aware are the regulators of these issues and uh, how long might it be before we start seeing some new rules? Yeah, so I'm not an expert in, uh, AI regulation, but I know there's a bunch of, uh, initiatives taking place both in the US and in Europe, uh, in trying to, um, provide those, uh, new laws.

Uh, I think even the, uh, AI companies are pushing for this 'cause they want to have some kind of adult, uh, responsible adult, uh, putting the laws in place to protect them, uh, from future, uh, lawsuits. Uh, but I think, um, it's tricky 'cause you don't want laws that will stop all innovation in on this front and this exciting new, uh, tech, uh, but some level of, uh, regulation is, is required and will be there for sure. Uh, part of it is covered, you know, with uh, GDPR and other privacy regulations, but the specific AI regulation is going to be required as well.

So ultimately, what's your best advice to folks as they kinda look at all these new pend toys? We're all excited about some of these capabilities, but, um, how do I do this in a way that, um, you know, is safe? Yeah, I I would say to, um, the average, um, you know, a fiso or someone looking to, uh, make sure is protected from the real world attacks, the top, um, priority attacks related to gen ai.

I think first, um, it's important to educate users on the risks and it could be in a, an automatic way or a semi-automatic way. For example, a perception point, we offer a browser security product, which is practically an extension to your browser. And when you're visiting any generative ai uh, web app out there, uh, it'll give you a message, um, explaining the accepted use policy in your company for using genai, um, and making sure the user and, um, approves before they use so such services.

And we also provide controls like before a user, uh, inputs some sensitive data into a generative AI web app or let's say open AI to GPT and many others. We show you, uh, a message if we detect you're about to submit, uh, sensitive data such as credit card numbers, social security numbers, customer names, and others into a gen AI web app. So this can practically help you, um, allow the normal good use cases of generative AI in the enterprise, like for generating marketing text and other things, but to be more careful as an employee when you're trying to, you're taking sensitive data by mistake and mixing it together with those gen AI apps that might not be taking care of it responsibly.

All right folks. You heard it here. Hey, the great news is AI is watching everything we're doing and it's gonna help us do it better.

Bad news is AI's watching everything we're doing. Hey, thanks for being on the show. Of course.

Thanks Mike. All right. And back to you guys in the studio.