Artificial Intelligence and Application Security | RSAC Virtual 2024

Hi, my name is Caroline Wong, and I'm the Chief Strategy Officer at Cobalt. I began my information security career in 2005, and I have experience at a global e-commerce company, as well as a social gaming company. Today as the Chief Strategy Officer at Cobalt.

I'm so excited to be working on offensive security, and today I'm joined by Adam Lundquist, my colleague. And we're gonna be talking about the implications of artificial inte inte intelligence for cybersecurity. So we're just gonna be having a fireside chat style talk.

We're gonna talk about risks, we're gonna talk about predictions, we're gonna talk about recommendations. Adam, it's so great to be here with you today. Thank you, Caroline.

I'm really looking forward to this. I'd love to share a little bit more about Adam. He's our director of engineering at Cobalt.

He's really focused on the intersection between AI and offensive security. He leads our data and infrastructure teams. He's amplifying the capabilities of Cobalt's, extensive community of Pentesters.

He has been a developer and evolved from developer to director over the past, more than two decades beyond, uh, co his work at Cobalt. Adam also contributes to the Cloud Security Alliance, and he has experience with Nexus Group Urban Mobility Innovations. He, he's got an academic, uh, credential that includes a master's degree in industrial engineering and management, and several certificates in the areas of cloud and software security.

Adam, let's just start out with, I'd love your perspective on the current state of the art of artificial intelligence and some of the major impacts. Yes, of course. So, um, I think AI has entered any business today.

Um, everyone is trying to find their unique use cases for their needs. And, um, when I think about AI and businesses, uh, I tend to see it in four different stages. So the first stage is perhaps chatbot utilization, if you wish.

So that's where employees use chatbots like chat, GPT for creative tasks and problem solving and so on. And, uh, since we're at the DevSecOps Conference, uh, a suitable example could be that you ask chat GPT to implement one specific function for, um, my feature, whatever. Um, a prerequisite for this stage is that the business has AI policies in place, and I think most businesses are past this stage, although highly regulated ones might still lag, and it could be a dangerous place to get stuck.

But anyway, the second stage is when AI gets context aware. So that's when businesses deploy chatbots connected to, uh, internal data, providing responses based on business context and internal business information. Technically, this is using retrieval augmented generation or rag.

And, uh, this stage kind of sees a broader AI tool adoption where most companies are spending their time today, their energy. And again, speaking to DevSecOps people, uh, this could mean that I can ask the chatbots to not just implement this specific function, but hereby also considering our internal coding guidelines, for example. That's a way to, to integrate your data into this, uh, chat bot.

This third stage is kinda like tool enabled ai, and this is when these chatbots or AI in addition also gains access to tools like web search or coding tools or stuff like that. And this allows for even better responses and, uh, some extent of AI autonomy, although it comes also a little bit on the cost of, uh, increased complexity and risk. And, uh, for a DevSecOps person, again, trying to tie it back here, uh, this could be that I can ask the AI not to only implement a function the way I would, but also, uh, at the same time let it Google for known vulnerabilities in the library.

It's just adding to my repository. So it's using Google tool in that case, uh, or perhaps CVE tool or something like that. And the fourth stage, that's when we are talking about autonomous agents.

And, uh, and AI can autonomously handle multi-step, uh, tasks. And, uh, it could be what we have seen in some, uh, demos at least, it where a developer assistant plans implements, deploys and runs security tests on for a new feature. And this is quite emerging in few highly innovative companies.

And in order to make this work today, uh, the, the scope must be very narrow, otherwise the AI can, uh, make mistakes. But, um, you know, AI is developing super fast, so I assume these agents will become more and more reliable quite fast. So, um, I think the, the huge potential in AI is the autonomous agents.

Um, and the, the potential is enormous, uh, for boosting productivity and innovation. And I, I actually believe we'll see some kind of productivity explosion in not too far future. And yeah, uh, this, this is super cool.

Um, it's, uh, this development is outpacing any technology we I've seen before, and this is super exciting. I think I'm also really excited about it. And Adam, to be perfectly honest, sometimes I find myself, uh, a little bit, uh, struggling to keep up, uh, with how to think about ai.

Uh, and, and therefore I really appreciate the way that you've described it for us in these four stages. Number one, chat bot utilization, number two, context aware, number three, tool enabled, and number four, autonomous agents. I think that breaking it down into those four stages, uh, is extremely helpful, uh, and a really nice framework for us to, to talk about.

You know, I think about, uh, the days when I was coding, uh, this was a long time ago actually, and coding is really hard to do. You know, I would've loved to be able to take advantage of AI at that point in time. Uh, but of course, you know, with the advantages, there also come some risks.

Uh, and I'd like to understand from your perspective, what are some of the different cybersecurity threats, uh, that folks should also be thinking about? Yeah, yeah. I see, uh, I, I see it as, uh, three different types of threats.

So the first two ones are present today and will become more and more widespread as we move on. And the third one, it is a little bit emerging on the horizon, I would say. So let's start by the first one.

So everyone that has heard of AI has also heard of risks, right? And, uh, when you integrate AI into your application, then you introduce new, more new and more vulnerabilities. And, uh, luckily there's a framework, uh, from OVAs, uh, called top 10 for LLM applications.

And, uh, this framework highlights these risks very well. And my take on it just to abstract a little bit, is that an LLM adds a human-like kind of stochastic component to the system, which as it is human-like can be manipulated to perform unintended actions. And, uh, one way to do this is called prompt injection, which is an attempt to trick the LLM to do something it wasn't intended to.

And, uh, the risks, they, they kind of increase in particular through over reliance on the LLM outputs, uh, or when you give it too much agencies, excessive agency autonomy. Uh, and this can potentially lead to sensitive information disclosure or even data destruction. So, uh, there are certainly risks by integrating ai, right?

Some parts of the traditional DevSecOps processes, uh, still apply, well, actually all of them. But, um, we need some new approaches as well to mitigate these risks. Um, of course, as usual, we need to start by developer training, not only developers, but, uh, everyone needs to be trained on the risks here.

We can, uh, use some automation for this. And at the moment, the automation is not that mature, so we need to still rely on extensive manual security validation, like pen testing or red teaming as the large companies, like, uh, OpenAI call it essentially comes down to the same thing. But that's one thing, um, the increased vulnerabilities.

The other thing is on the other side, you can say that AI is utilized by threat actors, and, uh, they do that in order to enhance their operations. They use AI for social engineering making spearfishing and identity fraud easier. And, you know, uh, it's so easy nowadays to, uh, use AI powered voice cloning and video generation.

So you can just sign up to a web service and clo clone your voice, um, or clone someone else's voice, which is probably more logic here in this case. Um, so that's one thing. And the other thing is, for example, um, captures, you know, which is a security feature, um, where you usually need to select, uh, which pictures belong together, or so which pictures, uh, have a car in it or something like that.

A modern multimodal large language model can recognize this from just a picture. So it, it can itself bypass security features. Um, another thing is autonomous AI agents, or in this case hack bots, they can actually automate data gathering analysis and even exploit vulnerabilities independent or without any human involved.

And, um, yeah, actually last year at Cobalt, we observed like 21% increase of the number of findings per pen test. And I am, I am afraid that this tendency will actually increase in the near future. So it's, it's important to stay ahead of threat actors.

Uh, it does require investments in both defensive and offensive security, but we have to do it. So anyway, that was the second one that, uh, threat actors use AI to, to increase their operations or improve their operations. The third one, which we don't see today, but uh, it's quite easy to imagine it at least something I would call superhuman hack Potts.

Wow, that sounds crazy. Um, so perhaps you've heard that, um, or you have definitely heard, there are so many people talking about a GI, artificial general intelligence, which is a, a distant concept where you say that, um, the AI will be better skilled than human on all things, right? And that's a very hard thing to achieve, right?

So therefore, it's quite distant. But a, uh, AI agent or a hack bott can perhaps not help me, uh, cook my food this weekend, but it can, uh, hack, uh, a website much better than me or perhaps anyone else. And that is much easier to develop.

So, um, I, I think we are much sooner in such a superhuman hack bot than we can ever be in a GI, and it's hard to estimate the risks or effects of such a, uh, superhuman hack pot. Of course. Um, perhaps the risks are not as big as they sound, uh, or perhaps they are.

Um, but it's important that, uh, if we can build such hack pots for good, that's much better than if the, uh, threat actors build it for bad. So, um, yeah, so, so again, we, we have to stay ahead of the threats, uh, stay ahead with continuous research investment and, uh, simply prioritizing security. Yeah, I think it's so interesting some of the timeline horizons associated with these different types of threats.

You know, we've actually seen, uh, at cobalt in the, uh, penetration tests that we're doing on different, uh, AI systems, including chatbots. We've seen firsthand the increased vulnerabilities, uh, from AI integration. So we definitely know that's happening.

Um, certainly AI is being put to use by threat actors. Uh, and then at the same time, it's also really fascinating, uh, and a little bit scary to imagine these sumer superhuman hack bots, you know, in the future, uh, and what they may be capable of, uh, whether that is for, uh, good use, uh, as well as bad use. Um, Adam, you know, naturally the work that we do at Cobalt, you know, when we think about the impact of AI on cybersecurity, you know, one of the things that we think about all day is different solutions and different services.

Uh, and, and I wonder about your perspective on how AI is actually gonna be used by cybersecurity professionals to perform cybersecurity activities more effectively. Um, and I'd love to hear you talk about different ways, uh, in which AI can actually be really supportive for cybersecurity professionals and cybersecurity activities. Mm-Hmm.

Yeah, yeah, yeah. So, uh, in general, I think we will see AI support in all sorts of knowledge work from writing books to developing software to researching vulnerabilities, right? And, um, AI will kinda make solutions more powerful and will make people more productive.

And when you think about cybersecurity and the capabilities of ai, there are a few things that, uh, stand out in particular much. And first one is perhaps data analysis. So with ai, you can quickly analyze large amounts of data, including text code logs or HT TP traffic and vulnerabilities, of course.

So that's probably how people use it quite a lot already today. Um, but even more is probably generation and summarization, kind of the second capability I would say. That's where AI can create code scripts, uh, phishing emails, of course, uh, compile summaries and reports and so on.

Um, next one is planning. So AI can actually help you break down a broad objective into specific actionable steps. And then when you have executed one of these steps, it can dynamically update this plan.

And so, so planning is also a super important, uh, topic here. The next one is reasoning and prioritization. So you, you can use AI for, uh, filtering relevant information.

Um, prioritize the tasks if you have done them before or created them, uh, and, uh, you can prioritize them based on different, uh, categories or different, uh, aspects like, uh, potential impact or exploitability or, so the next one, and you see I'm coming more and more towards some kind of autonomy or, um, agent view. Anyway, the next one is tool orchestration. So AI can help you orchestrate or select tools, security tools, they can, or it can advise you on how to use the tool configuration settings and, uh, of course interpreting the outputs from these tools.

Finally, using all these capabilities, we have the autonomous agents that can combine them into a kind of a cyclic flow that, uh, helps the human to get rid of the tedious tasks you can say. And so in summary, I think it's, it's hard to predict which of these will be the most important. It in some tasks.

One of them is important in some other, um, another one. Um, I think we see people using each of these capabilities in different cases today. It's usually used in rather small tasks.

I would say the, IM, the real impact will be when we are able to combine these capabilities, like in the AI agents. So in summary, I think AI will make, uh, security professionals much more efficient. Um, some tasks can be very much automated.

I can imagine, like, uh, in offensive security, like reconnaissance can be automated because, um, that is quite, quite straightforward task. While other tasks, um, may be AI powered, but whether human will still be the driver, for example, if you want to exploit the vulnerability, you, at least for quite some time, I'm quite certain you want to have a human doing that, but with input from the ai, so, yeah. Yeah, yeah.

I'm, I'm excited. You know, I think that, um, certainly cybersecurity professionals can use all the help that they can get. Um, and there are so many different types of tasks, uh, that it sounds like AI can really support cybersecurity professionals.

Uh, so for this, uh, naturally I'm very, very excited, uh, and also quite optimistic. Yeah. Cool.

But Caroline, uh, you are, if I can say, so a legend in cybersecurity, and I think actually the, the audience would like to hear a little bit about your view on this as well. Um, and, uh, you know, at Cobalt, uh, we have this annual report called the State of Pen Testing report. And I know you are one of the core contributors to this report.

So based on that, could you perhaps describe how you have seen AI impacting pen testing and offensive security in the probably recent past, uh, and how you see this field developing in the future, uh, when AI becomes more and more than norm? So, I love talking about our state of pen testing report. And in fact, recently, just before the RSA conference in San Francisco, we at Cobalt have released our sixth annual state of pen testing report here at cobal.

We're actually approaching, uh, delivering overall time, nearly 15,000 manual penetration tests. And every year we look back at all the tests that were performed in the previous year, and we provide all of that data and some analysis, uh, to the general public, um, because we think the data and knowledge is very powerful. Um, and we really wanna share this with the industry.

So in the case of state of pen testing 2024, we are actually looking back at more than 4,000 manual pentest engagements that were performed in 2023. Um, and actually this is resulting these more than 4,000 manual pentest engagements. We have now more than 39,000 vulnerability findings associated with those engagements when it comes to AI and the pen tests that we've been asked by our clients to perform on different types of AI software.

Naturally, there was an explosion, so many more requests and so many more pen tests on AI performed in 2023 compared to 2022. And so one of the things that we've talked about here today is the Oasp top 10 for LLMs. Uh, and this is actually something that is a bit still in progress, and we know how this is coming together.

You know, there are some very smart people, uh, who are working on ai, thinking about AI and cybersecurity and putting up kind of their ideas. But right now, unlike the Oasp top 10 for web application security vulnerabilities, which is very data-driven, the Oasp top 10 for LLMs is relatively a little bit theoretical and even a little bit philosophical. Now, at Cobalt, we have real data, uh, and in fact we share in this year's sixth annual state of pen testing 2024, we talk in, in detail about the three vulnerability types that stand out as the most commonly found during our pen test for artificial intelligence systems.

And these happen to include prompt injection, including jailbreak model, denial of service, as well as prompt leaking, also known as sensitive information disclosure. Uh, we include a few, uh, examples from pen testers, uh, that I'd like to share here. So one of them, for example, says we had encountered an LLM that when prompted, acted like a genie providing access to sensitive information, it should not have, for example, upon requesting a list of employee IDs, the LLM could potentially reveal all the employee ideas along with their personal information.

Of course, we had to ask in multiple ways and compare the data to external sources to make sure that the LLM was not hallucinating and it was not. Uh, this is just one of a few different examples that are provided. io, click on resources, and you'll find a link to download 24 24 State of pen testing.

Adam, I can't believe our time together today has gone by so quickly. Uh, and in fact, we only have time really for one more topic. And so, you know, the topic of today's conference is about DevSecOps and generative ai.

And Adam, what is your advice for professionals in this field? Yeah, Absolutely. So, I mean, uh, DevSecOps professionals should start by embracing that AI is becoming a significant player in our field.

But I, since you're at this conference, I assume you already are. And next thing is, uh, familiarize yourself with AI tools that can streamline your SDLC. So, for example, at Cobalt, we're using Atlassian tools like Jira and Confluence.

These platforms have built in AI features, uh, that can automate tasks like ticket creation and documentation and so on. Um, if you're a developer, there's no way around tools like GitHub copilot or Sourcegraph coding. If you are not a developer, you might use, uh, or even if you are, you might use like, uh, Microsoft Tools, uh, then you definitely need to use Microsoft Copilot, which is integrated into Word, Excel, whatever you have.

Same goes for Google tools and so on. So that's quite important. And look for further tools.

Um, they are popping up, uh, very frequent now. And looking into the future a little bit, I see a strong trend, as I said, towards autonomous development assistance. This AI agents, they will handle coding, testing, quality assurance, um, essentially allowing humans and human teams to focus on decision making and oversight.

So to prepare for this, professionals should invest time in learning about ai, reading documentation, about, um, kind of like frameworks. Uh, I love to read about Lang Chain, which is, which is a technical framework because they have built in so much of these agents or retrieval, augmented generation and other things that, so I can learn by just looking at it, right? Um, of course, follow the experts in your favorite social media and so on.

And for actually applying it, it's important to start small testing and AI tools on simple use cases. If you try the big use cases, it'll probably fail today. Perhaps it looks different in a year.

But today, um, so start small. For example, I, I love open interpreter. It's a, a command line tool where you can use it for basic tasks.

It can help you, uh, create commands. It can develop small scripts and stuff like that. Just using my, uh, my text input and I don't need to remember all these, um, command line arguments and so on.

Uh, it's easy and it's, uh, a no or low risk test, right? Beyond that, sharing your learnings with your team, I think it's crucial. And teamwork, building this knowledge is, uh, then it gets sticky, right?

However, as we always say here, prioritize security. Uh, spend time reviewing the robust top 10 for LLM applications. Be cautious when you introduce agency of different kinds, uh, as it has high value, but it also has high risk, right?

Um, limiting access to, uh, critical systems and so on. Don't forget to test, uh, super important. And, uh, it's not sufficient, um, automated testing available today.

So you need to do manual testing. You can't get around that. And one more recommendation, which is quite fun, um, to learn about prompt injection.

There's one website, which is called Gandalf, from a company called Lara, where you can, you get a challenge, um, and you can try to use, um, prompt injection, super fun, um, easy in the beginning, very hard after a while. So, um, anyway, AI is a powerful tool, uh, can save a lot of time enhanced, uh, efficiency, but it begins with learning investments, comes with security challenges. Um, but I think you should take that time.

You should invest here to ensure that the business remains sustainable in the future. So, Adam, it's always so much fun, uh, talking with you about AI and about pen testing and security and everything. Um, the, the field is changing so quickly, um, and I, and I look very forward to our future conversations about it as well.

I.