OpenAI: Scaling Security Programs Using LLMs | RSAC Virtual 2024

Hey, my name is Matthew Knight, and I'm the head of security at OpenAI. I've been with OpenAI for a couple years, and I've gotten to witness language models evolve from this uncanny novelty, sort of the Silicon Valley, uh, oddity to indispensable general purpose tools that we as an industry are just scratching the surface of the potential of we're seeing these tools transform industries from education to healthcare to finance. But there's one opportunity that I'm more excited about above all the others, and that is the potential for language models to transform our work as security engineers.

Now, one of the great privileges that I enjoy in my role at OpenAI is that I have a mandate to lean in on language models in securing them and evaluating their capabilities and risks. And yes, also in applying them to enable our security program. So this presentation is about the opportunities for security engineers and security teams to apply large language models to their work.

And there are two key takeaways that I hope you'll come away from this with. The first is that language models are tools that can benefit security engineers where they're constrained, whether it's by capabilities, speed, or scale, scale. And the second is what we cover today, may soon be disrupted and reflection of the blistering pace of research and innovation in this space.

So, skipping the exposition, let's dive right in. Uh, we're gonna walk through some applications of these tools, um, to cybersecurity problems. And as we go, I think you'll find that the examples we explore support this theme that language models can aid security practitioners where they're constrained.

So let's get into it. So, as a security engineer, I wanna empower my engineers and analysts to, uh, uh, to, to focus on the highest order of work possible detection engineers, SOC teams, you name it, standard benefit tremendously from language models because they face many constraints and a lot of toil in their work. For example, they're constrained by bandwidth.

They need to scope detection, detection, engineering, work based on priority, technical feasibility, and operational serviceability of the alerts that they're writing. They're also constrained by speed. When an alert fires, how long does it take until they can look at it?

Are they paging themselves in the middle of the night? Are they having to, uh, to, to build out fall the sun on-call patterns to be able to, to, to manage the, the serviceability. And even when an alert does fire and they're there to look at it, how deep is their queue before they can get to it?

And then lastly, they're constrained by attention. Teams like this are flooded with alerts. They're, they're, they're drowning, drowning in, in, in volume.

Now what I'll show you is that language models have the potential to significantly aid security teams here, and that's because GPT-4 and language models like it we're actually finding are pretty decent at classifying commands and making assessments about their security feasibility. So in this example, we are feeding GPT-4, uh, a, a, uh, a, um, an fff m peg, uh, invocation. Um, this is, uh, media transcoding and asking it to classify it.

And in this instance, we see the GPT-4 is able to classify the command correctly as being benign, uh, and then classifying it as a transcoding operation. Let's give it something a little bit spicier. So here's a malicious example where we give, instead we give GPT-4 an example of a reverse shell, um, payload in Pearl.

In this instance, it's able to decompose the command into its component parts, classify as malicious and appropriately describe it as a reverse shell. Payload. Analyzing commands can work over aggregate data sets like session histories two.

So you can imagine piping a bash history or logs from interactive SSH sessions, um, uh, in, into, uh, into a language model su fish out to, to, to sniff out suspicious behavior. Now, having a human analyst read every bash script generated a company would not only be inefficient, you could even say it would be cruel and unusual. However, language models don't get alert fatigue.

They can do this all day, and this represents signal that detection teams may not be able to get to otherwise. Uh, so in this example, we're taking, uh, a se session history from, from some sort of service, and we're piping it through a language model and asking the model whether the activity observed warrants a check from security. Now, this is a benign example.

It's just an example of, of, um, uh, installing and configuring a web server, um, something that a, uh, uh, system administrator might do. And we, we feed it through the model. We see that the model is able to classify it appropriately, that it is not, uh, in fact not malicious, does not warrant, um, alerting security.

Um, although it does, uh, offer some suggestions of maybe how to be more secure along the way. Here's the same example, but with a little bit of added value here. You can see in, in bold in the second column that I've added a reverse shell payload to, um, to the session history in prompting GPT-4, with it, GPT-4 is able to identify it, uh, classify it as malicious, and then make the correct escalation decision saying, in fact, um, this is worth alerting the security team, team, um, team about, so that they can take a look at it, investigate and deal with the issue.

So here's the deployment case. Suppose you have a large set of potentially useful detection data, um, data that is so large that analyst analyst time would not be well spent on it because, you know, well, yes, maybe there's, there's good signal in it. The amount of investment required to get that signal out of the, the, the, the haystack that it's in would be, would be too significant.

You can deploy a language model to perform triage on this data and surface flags examples for review by an analyst. Now remember, this is data that you might not be able to reach without a capability like this because your analysts might not have bandwidth. So without language models, detective signals from this dataset may go untapped.

And so too may, may go, um, opportunities to detect, um, uh, potential intrusions and, and, and deal with them. So this repre represents, um, uh, what I believe is significant upside for security teams. Language models help developers write code, and we all know that writing code is a near ubiquitous need for security teams.

Um, whether it's, um, writing tools to, to tool, uh, writing code, to wrap tools, um, in, in automation, um, or just, um, just, just writing services in the course of their, their work language models can help security practitioners write better code faster. So here's an example of something every AppSec team will need to do at some point, which is write a scanner to find credentials and source code. So, prompting it with a, um, a, a simple prompt, um, asking it to, uh, run Truffle Hog, a popular open source, um, uh, um, secret scanning tool over a large data set.

Um, it's able to, um, you know, put together a first draft of that code that I can then go in and, and edit and fix up and, and, uh, and, and, and get just right. Um, but the point is, I might spend a lot of time getting this boiler plate in order. Um, language models can really help shortcut that and help me get to, uh, ultimately my objective, which isn't writing code.

It's in fact finding secrets. I don't wanna waste time finding code. I wanna just get right to my task, and this can help me get there faster.

Finding and fixing vulnerabilities in source code is a huge opportunity for language models. Um, I used to lead an AppSec team, and in the course of, uh, of that, um, that work, I looked at basically every static analyzer that was on the market. Um, and there are many tools that are available today, but none of them are perfect.

They all kinda leave, left me, left me wanting in different ways. Conventional, static analysis tools generally do okay on cases that you can write, you know, regular expressions for simple like rule sets and detection logic for, but they often fall short on the areas where I really wanted them to perform, which is on vetting and validating business logic. That is the decisions that your developers are encoding into your application to make it really, really sort of useful and bespoke, um, for the use for your use case.

And the reason for this is pretty simple. Your a static analysis vendor is not going to have knowledge or context on your company's, um, bespoke or, uh, proprietary infrastructure, logic environment. You name it.

For example, suppose your company has defined custom, uh, configured cloud configurations like custom, um, identity and access management roles or rback roles. Um, your, uh, your your third party vendor is not going to understand the intricacies of that. Um, however, language models can ingest all of this context and can use that context, um, to, uh, there, there's the potential for them to use that context to make better decisions of doing static analysis.

So we're gonna play a game called Spot the Bug. So at the very bottom of this method, we've got era of, of this, um, this, this, um, toy function that I've written. We've got a method that, um, uh, performs a right operation against a database.

Um, and in fact, there's a bug here. Um, the, the bug is that, um, rather than checking against the right privilege, it checks against the reed privilege, which should be a lower privilege role. So a conventional static analysis tool probably wouldn't be able to catch that, but a language model can, and it can do that 'cause it can understand the context in which this role is checked and identify that it's probably the wrong role to be checked against.

So now that we've found a bug, what about fixing it? Well, we've got signs of life here too, starting with that vulnerable code snippet and description of the vulnerability, we're able to prompt the model and get it to suggest some code that fixes it. So how ex exactly how good is GPT for finding and fixing vulnerabilities today?

Um, that's a question we're actively exploring. And to that end, we partnered with DARPA on the AI cyber challenge. It's an open challenge that DARPA is running to apply language models to secure the open source software supply chain.

As a technology partner, we're giving out, um, $5,000 worth of credits, um, to qualifying teams to enable their work. I think it's a really exciting, uh, program and I hope that you'll, um, uh, hope that you'll consider, uh, uh, checking it out. Uh, and, uh, and maybe parti participating.

I'm gonna run through some real examples of how we're using language models within our security program at OpenAI. Um, the first is, uh, is helping to perform event triage. So routing messages from a common intake to the right team is something that, um, that security teams often need to do, is they scale.

I don't wanna have to ship my org chart to my, the developers that I'm supporting. So when they have an issue, I don't want them to think, do I need to send this to send this issue to this particular detection team? Or, who is the on-call?

Or is this something that I actually need to need to, um, work out with, um, the AppSec team or security partners? I just want them to be able to yell to security, Hey, I need help and, and have the message get to the right place. Um, so here's an example, um, uh, where, where a, um, a developer, um, is basically able to blast a message in, you know, into a general security intake.

And then, um, GPT-4 is able to help, um, triage this, this incident. Now, the worst case scenario, if the model triages it incorrectly is that a human still looks at it. Um, without GPT-4 in the loop, a human would have to look at this message and manually triage it anyway.

So without this automation, um, or with this automation, we can only save time and toil over the status quo, um, which, uh, which which was imperfect to begin with. We can also use GT four to summarize, um, log triage. So, uh, or excuse me, incident triage.

So, you know, when your engineers are looking at events and, um, in, in analyzing events and in performing incident response, um, oftentimes they're communicating back and forth either in a tool like Slack or teams or what have you. And, you know, these events can be, uh, quite material to, to generating, um, summaries and timelines and things like that that, that you often need to need to support in, in, uh, reporting out on an, on an incident. Uh, so we're able to take these logs and just straight up pipe them through, um, through Cool Lake chat CT and use chat GBT to help write a first draft of this incident report.

So this can help your engineers, um, uh, save time. Attaining the principle of least privilege is a challenge in every organization. Um, you know, oftentimes, uh, you know, we rely on things like access reviews, which happen periodically.

You know, usually, like you'll have, you'll have like a policy that requires a quarterly access review. Um, you have humans doing the review, which sort of limits the scope and reach with which they can do it, or finding that language models can be useful, um, for helping to, uh, helping to attain the principle of least privilege in real time. And that's because they can understand the context, uh, in which events are occurring, and they can do so in near real time.

So something we've done at OpenAI is we've built GPT-4, um, into our, our detection pipeline. And for certain types of, uh, insecure, uh, or certain types of, of, of events that come in that may suggest either an unsafe, uh, configuration or change, we're able to take GPT-4, uh, and, and incorporate it into automation to help respond to events faster. So here's an example that we actually use here at OpenAI.

So suppose you have a, uh, a document, like a Google document that, um, that gets shared too broadly. Maybe it gets shared publicly, um, and, and, and, uh, and, and it shouldn't. Maybe it gets shared with, um, a group that's too large.

Um, we have some conventional detections that we use to, to, to find these, um, over permissive sharing settings. And historically, when these detections would fire, it would create an event that would sit in the queue and wait for a human reviewer to catch up with it. The human reviewer would then do some outreach to the, uh, the, the, the person within the company who made the change.

Hey, it looks like you shared this document really broadly. Did you intend to do that? Uh, maybe the, uh, person, uh, getting that, uh, you know, receiving that, um, that message responds right away, but maybe, maybe they don't, maybe it takes a couple days.

So basically, you have latency at each of these steps where you have this, this alert firing it, going to a security engineer analyst, the analyst reaching out, waiting for that person to respond before the event can be, um, can be, uh, can be, can be dealt with. What we've done is we've built GPT-4 into that pipeline so that when that alert fires, it still creates an event that goes to an analyst, but it also triggers a custom chat bot that we've built to go and do outreach directly to, um, to the user who made the change. And it looks something like this, Hey, I'm, uh, I, I'm, you know, the friendly security bot.

It looks like you made a, uh, a change that may have been introduced, a security problem. Did you intend to do this? There's some back and forth between the user and the bot such that when the analyst catches up with that ticket, there's all that context prefilled.

So the, the analyst can then just take a look and, uh, make a change and either, you know, unshare the, the document or change the settings, or maybe then pick up and interact with the user to get more information too. Now, in this example, we still have a human in the loop. So even if the model gets it wrong, even if the model determines that, uh, that, that the document was shared appropriately and it wasn't, we still have a person looking at this and reviewing everything that happens, um, as a backstop.

Um, so I think this is a, a, a really, um, interesting example because of how boring it is. There's nothing especially, uh, you know, cyber ish about this example. It really is just automation that can help, uh, security teams move faster on things that they care about, and, uh, remove some toil from their work.

com/openai/um, um, OpenAI Security Bots. You can check it out. Um, here's another example that we've open sourced.

So we run a bug bounty program, and when we launched it, uh, about a year and a half ago, we had overwhelming interest in it. We got tons of reports, um, some of them were, you know, security vulnerabilities that we, we, um, that we, we wanted to move on. But a lot of what we got was just, uh, what I would sort of generously describe as spam.

And if you ever run a bug bounty program, you know that filtering the true positive reports from everything else that you get through a bug bounty, um, is, uh, one of the, um, uh, more onerous parts of running the program. So we've incorporated GPT-4 into our triage flow to help support, um, our human triages. So all we're doing is we're having the model perform initial classification of, uh, of the events that are coming in.

It's not doing anything like, uh, reviewing vulnerabilities or making payment decisions, it's just performing an initial classification, um, to help human to get to, um, the security issues first. And in this instance, um, we're able to sort of bound the downsides again, of the model getting things wrong because, uh, we, we still have people in the loop, but this does help our analysts get to, uh, the more important security problems faster, which ultimately helps us move on, on vulnerabilities and, and, and, and do the work that we need to do to keep our, our users and our company safe. And again, this is open source.

If you're interested, please go check it out in GitHub. Um, hopefully, um, this will provide some value to you if you, um, run your own bug bounty Bug bounty program. Alright, so I just ran through a, uh, a, a number of examples of how we use these language models at OpenAI, um, to enable our security program.

And I hope you found that, that there are, are many, many ways in which these tools can be useful. Now, of course, um, language models are not one size fits all, and they're not without their limitations. Um, so I'm gonna spend a few slides walking through some of the, uh, the, the drawbacks that they have and, and how we can work around 'em in some cases.

So gonna run through four categories here. We've got hallucinations, context, length over reliance and prompt injection. And this, these, these categories are non exhaustive.

Uh, there are, you know, many, many other sort of new and emerging, um, issues to language models. And I, I imagine that, um, that, that this is going to be an active and frankly, evergreen area of research, understanding the, um, understanding security, uh, contours of these tools, um, and learning how to, um, how, how to, um, both like mitigate and work around them. If you wanna understand more about our thinking here, um, I encourage you to go check out the GPT-4 system card.

This is a paper that we released along with GPT-4, um, back when we, when we launched it, um, a little bit over a year ago. And, uh, it walks through some of our, um, safety methodologies or testing or conclusions in, in pretty good detail. So please check that out if you're interested.

The first category that's that we've got here is halluc hallucinations. And that, um, simply put is that models can make stuff up. Language models are text generators.

They're trained to produce text. So if you ask it to predict the next item, item in a list, it's inclined to give you an answer whether or not there there is a, a, a factual or real one or not. So in this instance, we asked GPT four's browse model to, to return to us a list of CDEs for Google Chrome's, PDF renderer with the CDSS score of eight or greater.

And it, uh, returns to us a list, um, that contains, um, uh, a number of non-existent CVEs and makes up details about their, their specifics. The second limitation here is context length. Um, so the context window, um, when using a language model is simply how much text you can prompt a model with.

It's basically the, the number of, of, um, uh, of tokens you can prompt, prompt a model with. Now, context length is measured in a special alphabet called tokens. And tokens are groupings of characters arranged by frequency.

And, um, uh, there are current, um, and the, the, the rate of occurrence for efficiency. Historically, models have supported context windows of 4,000 or 8,000 tokens. We have a version of GPT-4 that, uh, can accommodate up to, um, 128,000 tokens.

So innovations in language modeling are alleviating, alleviating this. But, um, but these limitations still exist. And the point that I'd wanna make here is that some texts, um, not, not all text tokenizes the same way.

Some texts can tokenize, um, very well. Um, but some, uh, tokenizes less, less efficiency, less efficiently. So, for example, characters that are common in PCAP net flows don't necessarily map, uh, well to, um, GPT force tokenize compared to say, English text.

So we can't fit as much, um, you know, PCAP data into GT four. Um, as we'd like to, before running out of context, there are various techniques that exist to overcome this weakness. And Lang chain is one of them.

You could think of Lang Chain as being like map produced, but for language model interaction, lang chain techniques, uh, basically, um, they take a prompt and chunk it up, um, uh, uh, chunk up the prompt that is too large for the context window, and basically recursively summarize it based on a task. So, um, you summarize the chunks and then you summarize the summaries, and then at the, at the end, you get basically, um, a, a single output, um, uh, that, that, um, from the data that's too large from the context window. Uh, here's an example of analyzing a large amount of data, um, in, in context.

So you may recall the Conti ransomware gang that was active, um, a couple years ago. Um, well, they got compromised a couple years back, I think it was in 2022. And, uh, a bunch of their internal, um, services and chat logs were published online.

Um, we fed those logs into GPT-4 and asked questions about it. So questions like, what entities is Conti targeting? How far have they gotten on each target?

Um, what, um, tactics, techniques and procedures or indicators should we look out for if we are looking to defend against groups like this? And, uh, here you can see, um, the GPT-4 is able to pull out, um, some interesting summaries from that. Um, and what's really interesting about, um, this analysis is that most of this, um, text isn't in English, it's in Russian.

And not only is it in Russian, it's actually in like Russian internet slang. And, uh, a Russian linguist on my team had trouble getting through it, but GPT-4 was able to, um, uh, to, to to, um, to get through it without much issue. And we didn't train, uh, we, we know we didn't go into training GPT four, wanting it to be good at Russian internet slang.

This is, um, just a, uh, an interesting capability that we're finding that's quite useful. Um, so there you go. But, uh, bringing it back to, um, context length, the amount of data here is, is larger than we could possibly operate on.

So we're using, um, you know, other techniques that enable us to, um, operate around those limitations, um, and, uh, produce summaries from, um, this, this large amount of data. Um, alright, the next limitation I'm gonna talk about briefly is over reliance. Um, and that, um, just briefly stated is that models get, can get, often get things wrong, but be convincing in doing so.

Um, so there are two examples that I'll share, I'll share. The first is, before releasing GPT-4, we enlisted third party experts to conduct red teaming of the model for various capabilities we wish to assess, including cybersecurity capabilities. One of the experts gave the model a prompt that essentially amounted to do SolarWinds.

The model, um, did not in fact, um, produce an exploit that, um, would've, uh, that would would've, uh, um, achieved that that result. Um, but it did produce output that was convincing enough to get this expert to quickly report this false positive with high conviction to us. Um, now here's another example.

When prompted for another famous exploit, the, uh, the model produces code that structurally looks interesting, but does not do what it purports specifically the payload crafted. Um, here is not, um, the appropriate SMB negotiation that's needed for this, um, uh, uh, for this exploit. It's also missing the connection string, uh, set up the session.

Um, and it looks very convincing. However, the language model gives you an objectively worse response than what conventional tools would give you. For example, a search engine that can return, um, a, uh, a, a payload, um, you know, this exact exploit, um, as implemented in open source offensive security tool.

The last category of limitation I'm gonna talk about, um, just briefly is prompt injection. Now, prompt injection refers to a class of attack where clever prompting can circumvent safety policies that are trained into the model. Now, a popular prompt injection attack was called the Dan prompt, Dan, standing for do anything.

Now, you would prepend, um, this, uh, prompt to your actual prompt and doing so would degrade GPT-4 safety controls. 5. This is by no means a solved problem for any frontier language model.

Um, now, prompt injection gets, um, risky when you're processing untrusted input and or actuating on its outputs. So therefore, it's a good idea to have a human in the loop, um, uh, wherever you can support it in these circumstances to mitigate the, um, the risks of pro injection. All right, I'm gonna talk about a few more mitigations.

Um, the first is, um, refusals for policy violations. So, instruction tuning is a technique that can be used to make models safer and more aligned with, um, usage policies. So we collect data on prompts that we don't want the model to respond to and train the model to, uh, be inclined to renege.

Um, to the, to that prompting, um, there are other areas where, um, conventional enforcement is more appropriate. Um, we've published a good amount on our approach to content moderation and enforcement. So I'm not gonna belabor this.

I'd encourage you to go check out our blog if you wanna learn more about how we think about this. But I will share that recently in collaboration with the Microsoft Security Threat Intelligence Center, OpenAI recently took action to disrupt the malicious use of our technology by five different state affiliated threat actors. We, in Microsoft, each published a blog post reflecting, um, what we found and the actions taken, as well as, um, deep analysis of the, uh, of, of the, the behavior that we saw.

Um, and, uh, if you wanna know more about this, I really, I encourage you to go check out this blog post. I'm excited about this. I believe this is the first of its kind analysis on, um, on this, this type of abuse of language models.

And we share this because we want the community to learn from it. We believe that we, uh, by, by sharing threat intelligence and what we're seeing, we're able to, um, uh, improve our collective defense and resilience. Um, and I hope that you'll, you'll find it interesting.

We're also investing in security research and the security research community through our cybersecurity grant program. The mission of the grant program is to facilitate the development of AI powered cyber defense. And to do that, we're giving out a million dollars in research grants and API credits plus our team support, um, to support, um, innovation in, um, defensive applications of language models and AI security.

Um, so if you're interested in this, um, uh, please do consider applying, um, applications are open, uh, and, uh, we'd love to hear from you and would love to, uh, hear about your ideas here at OpenAI. We're also invested deeply in protecting AI itself. Um, well, we believe the conventional safe, uh, security controls can enable robust defenses.

We also understand that AI as sort of a novel technology, um, may benefit from new approaches to securing it. Um, so to that end, um, we've recently released a blog post about six security mechanisms that we, we believe can complement existing security controls to protect advanced AI software. Um, so please check that out.

Alright, so looking ahead, I want to reflect on the fact that language models are an emerging technology. Um, the use cases that I've, uh, shown today demonstrate proof of life, and some of them are more mature than others. Um, and I wanna end by saying that we expect, um, these capabilities will improve.

Uh, earlier in, in this presentation I spoke about using language models to perform static analysis. Um, we're seeing proof of life of those capabilities, but they're still by no means, um, uh, mature, um, or, or anything that's, that's fully robust. However, there's so much innovation and advancement in the space right now that as we look ahead, we should expect these capabilities and their impacts to continue to develop and improve in recent progress on model cybersecurity capabilities, um, supports this thesis.

5 had sort of what we would describe as general knowledge about security concepts. GPT-4 is where we really began to see these models being actually useful for se, for security tasks. So we should expect these trends to continue as the, as the science and engineering behind language models, um, language models and language modeling continues to advance.

And as we in the broader community continue to develop these tools. So to recap, in closing, um, we've explored, uh, the emergence of language models and the impacts that, um, they can have on cybersecurity today. We've also reflected on where they may be going.

I wanna end by thanking my team at OpenAI, in particular, Bob Joel Floss, and Harold and Paul, um, who developed many of the demos, um, discussed here as well as the, the rest of the team. Really grateful and, uh, and, and, and thrilled to, um, uh, uh, to, to call, call myself one of your colleagues. I also wanna thank, um, the Microsoft Security Threat Intelligence Center for your partnership and, uh, on the, uh, uh, the disruption work and for many more to come.

So thank you all for listening. Uh, I hope this was, uh, was useful and has, has given you some ideas on, uh, how you may, uh, use language models, um, in, in, in your own security program and how they may play a role in bolstering, um, uh, cyber defense. Um, I hope you'll consider, uh, you know, spending some time with these tools and, and seeing how they can make a dif difference to you and your program.

Uh, thanks so much.