AI Won’t Hack You, but it Might Just Get You Hacked | RSAC Virtual 2025

Hi everyone. Um, thank you very much for joining the Techstrong AI Summit. Um, my name is Dr.

Katie Paxton fit, and I'm gonna talk about, um, kind of where I see some of the big AI threats coming from. Quite often in the media, we've really heard about AI being like a force multiplier, maybe hack bots you've heard of, and a lot of people are worried about AI really coming to hack you. Um, now I don't know whether or not AI is really gonna hack you, but I think it might just get you hacked because there is a real growing challenge of securing the AI slash ml supply chain that I think are not enough organizations are really thinking about and talking about as they kind of go into the new AI era.

So my name is Dr. Katie Paxton, fear, I'm a principal security researcher at Traceable by Harness, and this is my AI threat top five. So these are the five risks.

So when I speak to organizations and how they're adapting to the new AI era, um, this is kind of some of the threats that I can see as a hacker myself. So the first one is really on de rely developers relying on AI generated code. So this could be your developers using tools like maybe windsurf.

You've probably heard a lot about vibe coding. So that's non-developers being able to use, uh, things like cursor in order to actually generate entire applications from scratch. Uh, a lot of the time now we're actually seeing that developers are not even professional developers.

They're maybe just, you know, they can be your sales staff, they can be your marketing team, they're not professionals. And with that we're seeing a lot of threats being reintroduced into Cobas. The second risk we've got is about data platforms.

Now, a lot of organizations nowadays really understand the value of data, and perhaps even more so in today's AI era that we've got, there are these platforms that ingest data and can give you a lot of insights into what that data shows and how you might wanna change your business based on the results. Unfortunately, with so much customer data, it makes it a real target for attackers. Uh, these platforms are often targeted.

Um, they are often, you know, people are getting things like phishing emails. It's one of the key ways attackers are actually targeting organizations. Thirdly, we've got AI tooling within organizations.

There's a lot of tools that organizations can now use in order to be more efficient or to really get the benefits of ai. And it's important to recognize that these two can be a risk of both a security risk, also a business risk for my, for my favorite, fourthly here is suppliers and suppliers and suppliers of suppliers using ai. You know, nowadays your security is not restricted to the brown bounds of your organization.

It encompasses a ton of different, um, products and services that you use. And while you may not be using AI in these ways, there is no guarantee that one of your suppliers isn't using AI generated code or using a data platform, and your customer data or your data as a company may be in there. And finally, it's how organizations are implementing AI into their products.

So thinking about, you know, nowadays every application has AI in it. What does that actually look like? What data is being set?

So let's start with number one, developers relying on AI generated code. Look, it's fairly obvious. Developers don't wanna write tedious code, right?

Developers often say they think AI to be kind of like a calculator. It's a tool that makes their job easier. It allows them to kind of automate very simple tasks that are just time consuming.

Or as someone on Reddit, put it, use it like an intern, let it do all the work. You don't, don't feel like doing, but check its work. And if you look at the stats, you know, people are using AI generated code more and more and more.

Um, this is becoming really, really common. So what can we actually do about it? I think everyone's first reaction is to go ban AI generated code like ban ai.

Stop it. You're not having cursor, you can't have windsurf, you can't have anything. I think it's wrong because I think that just leads to developers using AI kind of unpermitted by their organization.

Instead, I think you have other options. Invest in basic SaaS scanners in the build. Um, really make sure that you have got that in the pipeline with developers.

Give developers the tools they need to work. You know, give them the ability to use AI generator code. Give them ideas.

Now, developers shouldn't be using AI to write K or language they're not familiar with. Um, but at the end of the day, if they are, we can encourage the use of prompts that feature security and make sure when we are actually deploying that code that you know, it's, it's going through a proper peer review process. A lot of people will say they do, um, you know, your your typical reviews, code reviews, but in fact that's just someone seeing you looking over it and go, eh, it looks outright.

So really encouraging, you know, developers to take a step back and actually read their code and make sure they understand that. Number two, data platforms ingesting all of your data. I think this is very, very attractive for a lot of organizations because it allows you to put all of your data in there and connect and then get those insights from that data by just uploading it instead of needing to hire a data scientist or an entire data team or having to buy expensive software.

Instead, you can put everything into the platform and then use, create or use interactive dashboards, prepare reports, and they can really start to understand the trends. And one of the kind of key advantages here is that many of them are low-code, no-code systems. They are do not require, uh, experience.

They do not require expertise. They require an understanding of the data. And unsurprisingly, with so much data, attackers specifically target these.

Um, whether or not that's something like we saw Snowflake recently and it's gonna be your more typical, um, you know, malicious download link, run it, and then the malware gets executed. Or it's in a traditional kind of like phishing attack where people were explicitly looking for API keys for things like Snowflake. Data ingest platforms in general are huge targets.

They contain so much sensitive data and because they're used by a lot of different teams within an organization, often it's not gonna have a lot of security oversight or it may be completely invisible to security teams. You know, with info steal malware. This is where we're seeing things, right?

We've got malware that specifically target these data platforms, API keys, um, or even the accidental commit of API keys to these platforms. These are all being used as a way to get into data ingest platforms. Three AI tooling within organizations within an organization.

There is lots of different, uh, ways that people use ai. In fact, I would have the guess that most people listening here have had a meeting where an AI has joined in order to, um, take notes really common. So things like note taking applications, but also, you know, if you do a lot of long documents, a lot of people will throw it into chat GPT and say, Hey, summarize this.

Some people will use speech generation or they'll use it for content writing. And it's really important that when we're actually thinking about these tools, we want businesses to be more effective. We want to give people the tools that they could have.

And then it probably won't be surprising for you to hear that I'm saying don't ban ai. Again. I think, you know, it's very tempting to outright ban.

It's very tempting to go, you are not allowed to use this stop. The problem is, is that if you ban people, they're probably gonna do it and just not tell you. And that's far worse for your security.

As we say at traceable, one of the best things you can have is visibility into what your applications are doing. And the problem is if you ban people from doing it, they're just gonna do it sneakily. They'll do it in a way you don't have visibility on.

So set limits on things like AI note takers. You know, you should not have meetings where customer data is being shared or intellectual property is being shared. Um, and ensure that really whenever you're using a third party tool that you have those, um, SLAs in place to figure out what happens if there's a breach.

How are you gonna protect your customer's data? Next up we have suppliers and suppliers of suppliers using ai. Look, no application lives in a vacuum.

It doesn't matter what you work in or anything. A lot of people still kind of have this perspective that applications are monoliths and they just exist on their own. This is not the case.

No application lives in a complete vacuum. Every single one has dependencies, they have containers, they have builds. And you need a continuous solution that really understands not just, you know, a single application but an entire suite.

And this is particularly true for integrations. You know, when we talk about API security, APIs are used everywhere. This is not something where you can just do a search for API and just find it, you know, there are huge lists of public APIs that anyone can use and take advantage of, and APIs will use other APIs.

You've got this API supply chain. So even if you go, well, I'm not using ai, this isn't important to me. You don't know that maybe you are using one of these like APIs and they're using ai, you know, you have got a risk of that third party and it's really, really important that you know that third party and you are aware of that risk.

Um, and that's partly, you know, why having third party API visibility is so important. And finally, we've got implementing AI into your products. It is so easy to add ai.

You know, you choose your preferred ai, maybe you go for philanthropic, you go to chat GP t you go to pilot, you top up your account with some credits and uh, you implement like a really easy to use restful API. You can even get AI to generate it for you. You don't need to know how to do that either.

Or you can use an existing library built into the language. Um, one thing that we're really seeing is obviously the rise of agent ai. This is the idea that AI is not just a chat bot anymore.

Now it's able to act autonomously. It's to say, okay, you know, you ask, Hey, sorry, bye. Like I wanna fly to the us right?

And it will go and look at flights, it will look at hotels for you and then it will go and be able to actually go there and pay for it. And really the intention here isn't to do this securely, it's to do it really well. And that's a very difficult attack surface because you have so many different components.

You have the AI chat bot that you are using. You have your AI agents that are going and accessing, you know, the website for um, uh, the, the, uh, airline or the hotel. Then it's using maybe their APIs in order to connect to um, and actually make payments.

It's connect to a payment provider. Like this is not a simple attack surface. And certainly I think most organizations are very quick to jump on the, okay, we wanna have ai, we have AI everywhere.

We really understand the value of this. But it's not that easy when you do handle AI input into applications. Just do not trust the input.

Um, look, if you look at kind of the sensitive data that goes through, you know, I think customer data, right? It is crazy how many people just trust it. And if you look at the stats, the majority of people say part of what they use AI for is literally just understanding data.

Um, which is crazy to me because the AI can't necessarily do that in the way that, you know, a human can. So my advice, um, is, you know, perhaps unsurprisingly, uh, really look for when you see restricted data types. So this is like a screenshot from the traceable product.

So we do this automatically, um, but look for things like prompt injection, look for things like sensitive data. Uh, there's the OS LLM top 10, which is a fantastic project that really helps you understand the risk of uh, L LMS applications. But you've gotta always validate inputs, right?

You cannot trust what an AI gives you the same as you can trust a human. So TLDR zero trust is a philosophy is gonna be more important than ever in the next few years, right? It is more important than ever that not just your secure, but your third parties are.

And it is a must that you should include third party attacks in your risk assessments and security policies. If your security tool doesn't give you visibility into those third parties, it is not the security tool you wanna be using. AI isn't going away, it's not a phase.

It's here to stay and we need to figure out how to use it. This is not something that we can sweep under the rug anymore, right? We need to jump on this and make sure that we are being proactive.

If we get too caught up in AI security like jailbreaks, we're really not seeing the bigger picture because it's not just about jailbreaks, it's about the entire AI ecosystem. And five, ai AI infrastructure is gonna be targeted more in 2025 and beyond. And I'm gonna leave you on that really cheery note.

Uh, thank you very much everybody. I hope you have a great rest of the conference.