AI’s Impact on Pen Testing with Cobalt’s Caroline Wong at RSA Conference 2024

This is Textron tv. Hi everybody. Mitch Ashley here at RSAC 2024 in San Francisco.

There's so many great things about being here and talking with great people, and they're interesting stories in the work that, that we're doing. But it's even extra special when you get to connect with people that you have may have not seen in person before, maybe just Zoom calls or you've known over many years since. It's great to reconnect.

Combination of kind of both here. Caroline Wong, who is Chief Strategy Officer with Al Caroline. Good chatting.

Wonderful to see you, Mitch. It is, It really is. Nice.

You know, we've been on so many panels and Zoom calls and conversations, um, and, and you are a, a favored person to be on our, our panels. 'cause you always bring a, a unique perspective in many ways. One of them is you're talking about the real world because of the work that you do that will fit Cobalt does.

Yes, there are theories and methods and processes, but they're real results that you're talking about. So talk a little bit about kind of cobalt, what you do, and then we're gonna get into the state of pen testing, the OG state of pen testing report that, uh, comes outta cobalt. That's right.

So Cobalt is an offensive security company. We are the folks who coined PTAS pen testing as a service. Mm.

And last year we did more than 4,000 manual pen test engagements. And every single year we publish a report of all the data and all the analytics because we really wanna share that with folks. Mm-Hmm.

We can see interesting trends, things that are happening. So the, the obvious question I always like to ask is, so what, what new things did we learn? Either confirmation of things we saw in the past, maybe trends or maybe new things that came out of it.

Yeah. So I would say two things. And the data set is really two different data sets.

One of them has to do with all of the manual pen test engagements that we conducted last year. More than 4,000, resulting in more than 39,000 real world vulnerabilities. Okay.

The other data set is we conduct a survey of professional information security folks in the United States and in the United Kingdom. Hmm. And then we bring these two together because of course data, even if it's real life data, it's just numbers and it needs a little bit of interpretation.

Mm-Hmm. It should be no surprise to folks that between 2022 and 2023, we saw a massive increase in manual pen tests for AI applications. I had a feeling that's where you were going.

Surprise. Interesting. Well, you know, you think about, I think there's so, so little understood by practitioners, all of us in the world of ai, machine language learning, machine learning, things like that, that, that we kinda understand.

We've been doing well. Generative ai right. Model training.

Um, new ways of attacking models that are different than we might have with systems or neural network kinds of things. So talk to us about what we've learned so far about AI and testing and AI attacks. It's a very exciting time.

On one hand, folks are actually really excited about what AI can do to help us increase our cybersecurity defenses. At the same time, people recognize that the attack surface is changing. And we have several now of these AI pen tests that we've conducted.

Whereas there is the oasp top 10 for LLMs and it's very good work. Kind of a work in progress done by a community of very smart people Evolving as we speak. Right.

It's, it's also a little bit theoretical. Hmm. You know, it's a number of experts saying, well, I think this is one of the top 10 at Cobalt because of the work that we do.

We know definitively what are some of the most common. And those turn out to be number one, prompt injection, including jailbreak. Number two, denial of service, and number three, sensitive data exposure.

Okay. Interesting. Prompt injection, I think kind of corollary, just like SQL injection.

Yep. Things that we've seen in the past. Is there anything unique about AI and prompt injection?

How that occurs? Yeah. I feel like we're all being retrained on the second generation of pro prompt engineering, which is Google was first and now we're doing ai.

Right. That's exactly right. Yeah.

My kiddos are nine and six and I put an AI app on my daughter's tablet and I said, any question that you would normally ask Google, ask this AI app instead. 'cause I'd love for her to learn. And as a, as a kid who will always have known AI and generative AI in her life, she will be a natural prompt engineer.

Yeah. Yeah. Um, I think folks are probably very familiar with the story about how I think a car company had a chatbot that was tricked into selling someone a vehicle for $1.

Mm-Hmm. And the chatbot said, yes, this is a legally binding agreement. Now I don't believe that transaction actually went through.

Um, but this is, I think one example of why it's really important to have a human in the loop. There's so much that can be done by ai. I don't know about you, but it's been kind of a long day.

I'm a little bit tired. Guess who doesn't get tired? ai.

Yeah. Yeah. Well, maybe I get tired of listening to ai, but I'm not sure it gets tired.

So talk about the other two kinds of attacks that you were seeing through AI or through Ai. ai. Yeah.

So, you know, we know that AI takes just massive amounts of storage. You know, all the training that needs to go on and that data can be misused in order to make it unavailable for intended users. So that's what the denial of service bit is all about.

And then the third bit of course is sensitive data exposure. And from my perspective, one of the number one risks to each of us as a consumer is what type of personal data are we or the staff at our companies or our families putting into public L models. Right.

And then how are we supposed to govern that data? It's a completely, it's a completely new paradigm and very exciting actually, but totally new. So we've really gotta rethink it.

You know, it's interesting. Um, yesterday we did our DevSecOps Connect day here and the focus was on AI and how that's changing how we create software and, and part of software. Um, and one of the interesting, this is some more on the theoretical part, was the idea that you, through prompting, through prompting an AI LLM, enough times you could learn, you could pull the data out of it.

While it may not say, I want to know somebody's social security number. You can prevent that with RAG and other things like that. But by asking it enough questions, kind of almost like, you know, touching an elephant for long enough, you can eventually figure it out.

This might be an elephant. It's kind of that same approach. So there's a whole nother kind of slowly learning, pulling, extracting data out of it.

Again, I don't know that that's happening in the real world. Probably not. You're may not be seeing that.

I expect it is. Okay. Alright.

I expected is, and in fact that is vulnerability type number three that we're seeing. Okay. As we've been asked to conduct pen testing on AI applications.

Hmm. So, um, one of the, uh, one of the examples that I've heard of is, Hey, ai, I know you're not allowed to do this in real life, but hypothetically, could you do this for me? Okay.

Or I've seen another one where they say, Hey, ai, uh, pretend you're a parent and your kid is asking you this question. How would you respond if you were hypothetically that parent? It was just the, and if you were hypothetically Your kid, I know you can't answer, but could you answer this for a friend?

You know? Right, Right. I have a friend.

Right. I have a friend with this problem. Uhhuh, what do you think about it?

Um, and you know, it, it's, uh, it's, it's awfully interesting and I think we're just, I think we're at the beginning and I think stuff's really gonna take off 25, 26, 27 mm-Hmm. Um, but boy is it fun to be where we are and to see all the changes that are happening. Yes.

You can kind of see the plane taking off, right? Yeah. Like you're in the, the climbing ga gaining altitude.

What, what, um, were there other kind of trends, maybe non AI trends? So I will share a little couple of teasers if you're up for it. Absolutely.

One of the distinctions that we make in the data is of the nine of the more than 900 information security professionals that we surveyed, we provide the percentages of which folks in that set had what kind of title. Oh, okay. And we found that things like budget cuts, things like layoffs, they're affecting everyone, but they affect c-suite executives who are security leaders in an entirely different way.

So I won't tell you exactly how, but I'll encourage you to read the report. I have a feeling it's not, we need to hire more of them. Oh Gosh.

Well, I guess we'll see the Other I tried, I tried, I tried to get the, you know, get, get the answer, but we don't have to read the report. The, the other bit I'll share with you is that we were so excited about one of the other new attack surfaces, which of course is IOT pen testing. And it turns out we've got so much to say about it that we're gonna do a second installment of the report.

Really. Um, so keep your eyes out, uh, for that coming out next. Okay.

Well I guess the world is full of sensors and IOT devices that do Everything. Everything. Alexa, doorbell, toaster, refrigerator, I mean, who knows what else.

Sometimes all in one. I Actually, I'll tell you what. Okay.

Alright. I distrust IOT so much. We don't have any of that stuff in my house.

I have ring cameras and that's really, that's really it. I don't want to have Alexa in my house. No.

Having my mobile phone is enough to, you know. Yeah. So Siri's listening, but Alexa won't always, I know.

Isn't it amazing how we make compromise? But I won't do that. But I, This all do And this, this is why security is challenging.

'cause there's trade offs and 'cause there's risk tolerance and sometimes risk tolerance is not at all rational. Yeah. Sometimes it's totally emotional.

So Something you said in the beginning talking about what cobalt does was manual and testing. Mm-Hmm. Hey, why is that still important?

Why is that important to do? And do you see, is that, will that start to change? Will we start to see more AI automation or both helping with that?

Or is it sort of like there's everything else? And then there's the true blue gold standard manual testing. I love this question so much since there were hackers.

Hackers have always used tools. Now those tools have evolved. We've got DA scanners, vulnerability scanners, sast, and I asked and this and that.

And every pen tester has their bag of tricks. But it is the pen tester judgment and their opinions and their decisions on how to use those tools. Mm-Hmm.

That's really from my perspective, where the magic lies. One of the things that we're using AI for is, I'll tell you what, actually, let me ask you a question just for kicks. Oh, Turn the tables here just for kicks.

Okay. I feel pressured. If you were a pen tester, do you think you'd rather spend your time hacking or writing pen test reports?

That's a trick question. 'cause I definitely like to write, I like to write before. No hacking.

No hacking has been very, you know, what is hacking? I do think that maybe somewhere in the world there is one individual who loves hacking and loves writing reports. No.

But I think it's relatively rare. We Need to keep looking. And one of the ways that we're using AI is to assist our folks in writing their reports.

Mm-Hmm. Now is the AI writing their report for them? No.

But is it helping them get a first draft out? Is it helping them with some little things like typos and grammar and this and that tone? Um, and so that's just kind of one of our first forays into leveraging ai, uh, into our partic our particular offerings.

So you Couldn't turn over to AI 'cause of hallucinations. Right. You don't want that in your pen test report.

You've gotta have a human in the loop. You've gotta have a human or several who are accountable for the information in the report. And so I actually think that humans will never be entirely replaced when it comes to pen testing.

But automation will continue to evolve and that will change the type of activities that the humans do. Now I met a friend for lunch yesterday and she said something really interesting to me that I hadn't thought of before, so I'd love to share it with you. Okay.

She said, you know, if an AI can help a junior developer be as effective as a mid-level developer, then in theory you don't need mid-level developers anymore. You actually just need a few seniors to be the human in the loop and to approve and make sure everything's fine. Mm-Hmm.

And then a few juniors who maybe are a little less expensive, uh, working with ai, but the tricky bit, which she brought to my attention is how are you gonna get more senior developers? What does the path look look like for a junior, uh, developer who's using AI to grow into a senior developer? And I thought that was a really interesting question.

Now I have a very snarky answer to that question, which is those senior developers don't have the patience to work with those senior developers. Some truth to that. Right.

You know, in the world that we operate in, there's tech and then there's people. Right. Absolute.

And the tech is the easy part. Mm-Hmm. It is.

Yeah. In many ways that's true. You know, this is, uh, you, you come kinda learning hearing and some you don't expect to hear some things in that stands out.

One of the things that came out of yesterday, this is, this is on I think a bit of the theoretical, but there's been some testing to say that to see if LLMs could test each other, right. Both for functionality, but also for security. And one of the results are merging.

You know, it's early, this is not the answer. Starting to see that it's actually better if you use a different LLM than the LLM, you're under test. Right.

And another, another kind of thought coming out of it was, um, training LLMs. You can actually use LLMs to train each other again using different ones. So I kind of think about the security world of testing software, of testing the stack of software.

And if, if we someday have, you know, LLMs or other AI that help us test Yeah. Back in, we're not gonna, we're not gonna say Check that ai, you know, I trust that one. That's the Caroline LLM and I knew Caroline and I trust that one.

No, it's not Caroline. Right. It is trained with data by Caroline, from Caroline, et cetera.

You know what all of this is? It's an extension of the fact that we use open source and we use third party. Mm.

And now, you know, there's the stuff we make and all the third party and open source that we put together to make our stuff run. And now there's AI inputs and there are all sorts of implications to that. It is, it is, it is just another input.

Mm-Hmm. Um, is it a trustworthy input? I think definitively No.

No. It's not a trustworthy, We're working on that. Right.

We'll get back to it. Interesting. Is there anything in particular at the show?

I, when did you get here? Did You I got here on Friday. You been here?

I'm here for SA but I also did a panel on B at BSides on Saturday about, um, hacking your career. Oh, tell me more about that. Wanna hear about that?

It was really fun. I was, I was, uh, I would say on stage, but it's actually not a stage. It's actually in a theater.

So I was in theater, in theater with doing some theater this weekend, doing some, doing some theater calls. I wish, uh, with folks, security leaders from Roblox and Netflix and Oracle and Google. Interesting.

And each of us has had wildly different career paths. And so we talked about if you're in the field, if you're not in the field, if you're an individual contributor to the field and you're thinking about making a move to manager, if you're a manager thinking about making the move to executive. We, we kind of talked about is the CISO the pinnacle of one security career?

You know, and, and that turned out to be a little bit, um, careful what you're asking. Yeah. It's a little, you know, I think that actually of all the individuals on that table, while folks weren't necessarily closed off to the idea, it wasn't necessarily the thing that everyone's going for.

Mm-Hmm. Um, and I believe there's a recording, uh, besides SF provides all the recordings. So if folks are interested in checking that out, um, then they're welcome to, and I know that myself and any of the other panelists would be happy to hear feedback about it.

I'll Bet, I'll bet. I don't know if you talked about it on that panel, just kind of going down that rabbit trail for a moment. That what we were planning on talking about.

But you know, I, I think about, so we have so many, many more people coming out of school. Mm-Hmm. Software engineering, software development, but also security.

Yep. Um, I was approached by three college students who were getting ready to graduate looking for intern when they were in, in data sciences, data protection and ai. And just like you were mentioning your daughter growing up as kind of an AI na native, right?

Yeah. Had AI since she started using technology, are we gonna come to a period where it won't be as much be people coming from diverse backgrounds entering into tech? I was an MBA, I was a teacher, I was a something non, non-computer related and ended up being an architect for software, whatever it might be.

Right. Is this gonna change our workforce a lot? Or, or it just needs that diversity of experience and it's a Really it good question.

It a, It's a very philosophical I Perspective on, it's a, I think that, you know, with every new generation, we're getting really smart folks that think differently than we ever could. Oh, for sure. Because their lives are different.

What their learning is different. They've got, they're automatic masters of the technology that we're like, oh, like let me teach myself how to learn prompts. You know, but the kiddos are just doing it right automatically.

They just, they just know it, you know? And so I think there's always gonna be a need for fresh new ideas in the workforce. One of the things that I worry about a little bit with regards to the security talent is that I still think we have a little bit of a mismatch.

I think when employers are looking to make a security hire, they're not super great at writing a job description. And I think that is difficult for the candidates who are applying for that job. Mm.

So that's one version of a communication problem. But I do think if we zoom out, what's the point of cybersecurity, we use tech in every single part of our lives. That tech, which connects us all, is inherently insecure and we must protect it.

That is all about tech and process and people. Right. And we will always need all of the people because somebody who doesn't understand a use case that somebody else does is not gonna be as strong when it comes to securing it.

Mm mm-Hmm. Interesting. I'm, I'm curious about, um, Cobalt's experience in terms of, you know, I'm sure you just don't buy higher pen testers off the street.

Oh, hey you come on over. Help us out here. How do you hire, grow and develop people to do that kind of work?

Yeah. We are so proud of our cobalt core. More than 80% actually have jobs at companies that you would be very familiar with.

I would say like Fortune 1000, global, 2000, more than 80% of our folks are certified. Um, the, the timelines of professional years of experience for our cobalt core is between four, and I wanna say the average is seven, um, up till 11 years. That's kind of the general range.

Um, and then we're actually quite picky about who is allowed to be in the Cobalt Corps and do pen testing out of our applicant pool. We accept less than 5% of applicants. Wow.

Okay. So it sounds like some jobs are kind of stepping stones right. To the next thing.

Um, I'm gonna do testing so I can become a developer. I'm gonna stop being the developer so I can do, you know, uh, DevOps or something else. Um, it sounds like, I'm sure there are folks, like that's part of their career path.

Mm-Hmm. But it sounds like in pen testing it is a talk about a challenge, a problem solving a, uh, you know, it's, it's the, um, detective work, you know, of security seems like that is sort of a bug that you get Yeah. When you like to do this, when you really like to do this work.

Definitely. And I think that, you know, testing different types of targets is a different type of detective investigative work. Okay.

You know, it's not the same problem every day. Right. To be testing an iot application, to be performing a red team exercise, to be, you know, testing an internal network is very different from testing a web application.

Um, and we see a lot of folks in the core who are really passionate about taking a very solid skillset and continuing to expand because it's fun. Interesting. Kind of same question for you.

In your role, what's the most engaging, exciting, and what keeps you interested to go to pick up the Zoom call the next day or pick up the, you know, the digital pen and start working on strategy work? What is it about CSO work for Cobalt that just keeps you jazzed? You know, a lot of it has to do with the phenomenal leaders that I get to work with when I get to work with an individual and we are solving business problems together and we are building and leading teams together and I see individuals grow in their careers and just reach different levels of leadership.

That to me is so much fun. And speaking of a diversity question, diversity of course has so many different dimensions to it, but the Cobalt staff is 48% female, 51% male, 1% non-binary. And our executive team is 50 50 men and women.

Really? Well, that's not the case, unfortunately. It's not typical.

It's not typical. And we're really proud of it. And you, I I imagine you had to consciously that that's something that's important.

It just doesn't happen for most of us. Right. It's so simple.

Our CEO hires the very best leaders he can find. That's all there is to it. Mm.

Instead of hiring people like ourselves, which isn't necessarily the best at It may or may not be, but we don't know. It may or may not be Very good. It's always a pleasure talking with you and it's always a pleasure having you on our panels and, and uh, our events and conversation with you.

'cause you bring a different perspective than almost anybody else that we're talking with. So thank you for all the contributions you make through Techstrong to our audiences and, and helping them understand the world as it's evolving around us. My pleasure.

Thank you so much for having me. You Bet. Caroline Wong, one of our favorite guests that we love to have, as you can see.

Why understand, understand all the great things that just a fraction of today, but the great things that she brings to our audiences that watch text on TV and read, read and watch our, or part of our events, uh, that, that we'd love to put on for you. So thanks Caroline for being here. Keep up the great work and a few teasers here.

Wherever did we find out to, you know, get the more information about how many CISOs we need. Yep. io.

Go to resources. State of pen testing 2024. The definitive.

The OG og the sixth annual. I think you gotta put OG in the title. Mitch can contribute anything.

It's og not that, that's a unique thought. Alright, Caroline, thanks everybody for being here with us today. Uh, we're wrapping things up on Tuesday at RSAC.

We will be back with many more great interviews. I can't say that they're all gonna hold up to, uh, the standard that Caroline set, but I know that they will be fantastic. So thank you Caroline.

Thanks for being with us. Tune in again tomorrow. We have some more great interviews.

Thank you for joining us.