Al and Security – Securing Al-Powered Development and Applications | RSAC Virtual 2025

I have a great pleasure of being joined by a wonderful panel here. We will make sure that everybody's mic is working. Steve, would you introduce yourself?

Sure. Happy To. Uh, hey everyone.

My name is Steve Boone, uh, executive of Product Growth at Check Marks. Uh, excited to be here and, uh, answer your questions. Wonderful.

Yeah, Naomi, Excited to be here. My name's Naomi. I work at Contrast, the best security company ever.

Best one on the stage, right? Well, my Boss. My boss is Oh, I see.

Okay. Right There. Uh, but no, I really do love it.

I am the senior Director of product security. Happy to answer any questions about that. Fantastic.

Tyler. Hey all. I'm Tyler, head of product at, uh, Sonatype.

Nice to be here And fantastic. And we have a special substitution at the end here. The slide is not caught up with, uh, Romy sauce.

Who's joined us. Welcome Romy. Hi.

Yes, thank you. Hi everyone. Uh, Rami, I'm one of the founders and CEO of, uh, men filling in for Baal because this was originally my project before Baal came in.

So Good to have you all here. Let's start out with kind of where we are with AI powered AI assisted development. You know, there's a lot of data out there, whether you kind of intrinsically watch LinkedIn or, uh, TikTok.

Somebody talking about vibe coding and what they're doing with the latest development tools. Or for example, we've done our own research just around how many people say that they're using AI tools in development, saying where between 40 to 60%, that could be AI assisted code completion, it could be vibe, coding, et cetera. I'd love to get each of your sense of where are we of thinking about security and all of this, because of course, we've never rushed before and tried out new technology before we thought about how we were gonna secure it.

So, you know, we'll know to do it the right way this time. Uh, maybe Naomi, you wanna give you a start and give us your, I'm gonna pick on you, uh, if you wanna give us your, kind of your sense of, are we talking about security in terms of how we create software with AI power tools? I think actually this is like another, it's an old story told again, because if you think about who's old enough to remember the first time you ever used open source or your developers begged the security team, can we please use open source in our projects?

And you're kinda like, Hmm, I don't know yet. And now you look back, you're like, well, that was hilarious. 'cause everyone uses open source nowadays.

I think we're just experiencing the same thing. An old story told, again, the use of ai, our developers are using AI begging security teams. Is it okay to use AI in 10 years?

People are just gonna be like, how did we ever try to stop you? It's just that story told again. And I don't think our job is to, to say no.

I think we're gonna try to tell them how it's always the purpose of security. And the fun part is, you know, playing alongside them, making sure they're doing it right. Um, to answer your question directly though, right?

Actually, I did forget your question, so I'm just gonna keep talking. The question was, are we talking about security as we're using AI Powered? We are because the developers are gonna be doing it without you, with or without you.

So if security is alongside them, play nice and making friends, they're gonna start listening to any guidance that we can give them. What we're trying to figure out now, it's a little chaotic as a security, uh, industry, we don't really know what we're talking about yet because we don't understand what they're building yet, understand what they're building, it'll be a lot easier to give them guidance. I, I really like your analogy of kinda the open source usage.

I think, you know, when we think about open source in your organizations, my mind goes to, uh, supply chain security, right? Understanding what's coming into your organization, especially if you don't take ownership for it. So AI is no different, right?

If our developers are leveraging ai, uh, the best time for us to be looking and analyzing and scrutiny, that code is at the moment, moment that they're actually generating, right? So if we're gonna encourage our developers to leverage AI as a new powerful tool to help them generate faster code, do we also need to support them with the right tools that allow them to scan that code, get instant feedback on the quality of that code, and give them confidence on whether or not to use that code? Because it's not a silver bullet, it's not a magic wand.

And so really our job now is to start putting those right guardrails in place so that organizations can use AI more confidently. Yeah, and I think that there's two vectors there, probably more than two. There's the coding assistance you talked about.

Mm-hmm. And then there's the AI models and, and technologies that being put into the software that the developers are creating themselves, right? So there's the kind like, uh, you talk about open source components, like a component selection.

You're making a model selection. So there's that vector and then there's the vector of using the, uh, coding as systems and toolings. I think we, uh, it's our responsibility and privilege to work in both areas.

Uh, I think the coding as systems get more press, um, but I don't know if they're more important or actually less than what's being brought in to the, uh, data science pipelines and product delivery pipelines in the products that are being shipped and, uh, provisioned, Right? I think there's also the issue of, uh, of scale, volume of code being generated, right? So with automated tools, you can, one person can create, I dunno, three times, five times, 10 times more code than an individual contributor typing on a keyboard.

And so that's almost by definition going to cause a lot more vulnerabilities to occur in that code. That's one. Two is almost all of these models have been trained on open source corpus, which inherently is less secure than what you'd find on average in an enterprise that goes through some testing and some QA and some, you know, rigorous, uh, process of release.

No way. No, No way. Yeah, I've seen it.

I've heard it Read About that happening. Yeah. So, you know, I think everyone expects code generated by an agent to be less secure, uh, on the one hand.

But to Naomi's point from before, these people have been around for open source 10 years ago, 15 years ago, and are not as resistant now. So they get it, they learn the lesson, they're not going to try and stop the tide with two hands, and they're already starting to think about, you know, the right way to adopt it and not resist it. Yeah, I was thinking about the same thing.

Different analogy though. If you remember cloud, we're not going to the cloud 'cause it's not secure. How long did we kind of stay in that one or two year period before we started really moving applications and software there?

That's Actually very interesting analogy for a different characteristic of, of ai, which is, you know, when cloud came out and everyone rushed to protect it, it took it a long time before it got into a meaningful market share of the number of applications in the world, right? So within one year, two year, three years, 10% of applications were on cloud. And I think it's the same with AI now, where less than half a percent of applications in the world today are ai, but the most innovative, the most strategic, the most future looking ones are on ai.

So I think it is worth watching, uh, what's happening there despite it not being that of big, of a portion of the world yet. Yeah, I I it's definitely arguable that we're moving a much faster pace in terms of adoption to AI and our tool set, et cetera, than even cloud was. I'm curious, we're in a different place too of where security teams and software teams are today and how they work together, the kind of tools that they need, uh, collaboratively defining what those are, selecting vendors, et cetera.

Is that part of why we don't see security teams in enterprises saying, wait, wait, stop, don't be using the stuff. It's not secure. We can't, we haven't secured it yet.

So we don't see the brakes being put on. At least not, you know, to the degree I'd argue cloud was Steve. Yeah, I, I think there's a really big opportunity, and I think that's what a lot of AppSec leaders see, right?

There's an opportunity to use AI to get your developers engaged in AppSec, right? I mean, let's face it, a lot of developers aren't security experts. Um, but now we have the ability to bring them the information they need when they need it, about a vulnerability to give them the, uh, feeling that, Hey, I can roll up my sleeves.

I can actually start to work on this. I can remediate this. I can, I don't have to spend days looking at it and trying to debug it or learn about it.

I can position a developer, you know, I always make the analogy of running a marathon. I don't run a marathon, right? That's not my, that's not my, uh, skillset.

But if you put me two miles from the finish line, alright, I can do that. And I think that's one of the reasons why we're seeing this, uh, you know, both AppSec and developers embrace this, is it's giving everybody this headstart. It's putting people right in front of their finish line of their goals.

And now all of a sudden, even if I'm not a security expert, I got the right context, I got the right confidence. And if we put in the great guardrails, now I feel like I can go and actually be engaged in AppSec. And by the way, if your developers are engaged in AppSec, that's how you're going to have a faster and more productive AppSec program, right?

You need them to be engaged. But isn't that so possibly dangerous stuff? You're starting two miles before the finish line, those first 18, how many miles is in a marathon?

2020. There We go. I've never run one either.

Yeah. So those first few miles, what, what if something had happened in those first two miles? I twisted an ankle, I hit a pothole.

Sure. Or I did something and now, now I'm hobbling towards the end because I never had that foundation, right? I think we're setting ourselves up for a potential failure in the future if we rely on AI too much as a crux.

That's what I say. Well, I I wouldn't say use it as a crux, right? There's, there's gotta be a, a balance of, of trust, put, verify.

Um, but at the same time, if we ask, you know, an average enterprise, how long it takes to remediate a vulnerability? If you don't know that, work on that. Now, when you have that number, how do we improve it, right?

How do we improve the mean time to resolution? Well, AI can really help us as a developer if I don't have to spend two days on that and I can get that down to a couple of hours or even the time that I might research to understand the risk to actually come up with working code, that's a huge benefit. Like that's worth the risk in a lot of places in the trade off.

Again, you gotta have the right checks and balances in place, but I think that's ultimately where people are embracing this technology is that it provides a lot of opportunity. I also think you have a growing with a difference is you have these kind of, these more security as code more security groups, writing code platform engineering teams that are responsible and want to use the tools themselves. So there's this, how can my life be better?

They're, they're seeing that the code that they're writing themselves is, uh, faster and more efficient. So I think maybe than a, a decade ago, you see more or security as code security coding. And so that kind of wanting to embrace from the beginning, I think is, you know, kind of raising the floor of expectation and velocity.

Alright, so look, honestly, I think we just haven't had an Equifax level security event coming out of AI yet. And so I, I think it's a question of when, not if and when it happens, that's when you know how good the relationships between security and developer really are, right? I think right now it's kind of up in the air.

Some, some organizations are very good at it, others not so much. Sometimes you see friction, sometimes you see a lot of push and pull. Uh, but I think it's still to be, to be figured out what it really looks like when, when the rubber meets the road.

So it's, uh, during those trying conditions, everybody reverts to type, right? It's, we all work well nice together when under pressure, right? It's a little tougher.

Um, by the way, we'd love to have some questions for our panel. If you wanna step up to a microphone, I'll do my best to to call on whoever steps up if I'm not blinded by the, the lights here. But feel free to join us with a question.

I'm curious, all, all of you talk with security teams, development teams. What are the questions that organizations are asking you about using AI tools, the security of the software that we're developing, maybe even the security of the tools themselves, right? These, a lot of new tools that we're using.

Uh, what are customers bringing up to you or potential customers asking you about AI and software development, Rami? Sure. So I think, again, taking on the, uh, corollary of, uh, of SCA and open source, we are super early days in the, in the lifecycle of ai and most people I talk to just wanna know where they even have ai, right?

So if you talk to a large enterprise organization, mostly we talk to, you know, the security people, DevSecOps, compliance, those kind of people, they have zero visibility into what the developers are doing, right? Very commonly, I've had hundreds of conversations where I'd go and ask again, our champions, you know, how many applications do you have with AI in them? They would tell me, you know, 10, 15 and then run a super basic simple scan and find 200, 300, right?

So I think the first kind of stepping stone is for them to just figure out what the engineers are doing and where and to what extent before they can even start any kind of, uh, meaningful analysis of security or anything else. That Inventory discover stuff, right? Right.

You need to know what you have to know, what you have to protect. Exactly. Well, it's even more hard because no one's making their homegrown models.

They're building off philanthropic, they're using Claude, they're just taking someone else's work and then building on top of it, whether it's an MCP server that they've built for themselves or some integration or some tool. So they're, that's just another thing. If we're gonna be doing asset management on ai, we have to know exactly how the developers are using ai.

So they're borrowing from these places. How are they actually using it in the tools that you're building and the stuff that you're using for your own customers? Like, what, what is going on in their ecosystem?

And I think it's gonna differ between teams. You have to have those relationships. I would echo we hear the exact same thing verbatim.

We also see an interesting kind of upshoot or, uh, kind of the first green right outta the ground of, uh, those same people now being asked to also look at the data science organizations, at the data engineering or organizations. Mainly we see as a lack of, well, we don't know where else to put it. So kind of feels like AppSec, even though we're not shipping the product, we're deploying into production.

So we also see that same sphere of scan and understand in the AppSec teams are now being asked to also look at data science teams, which is a whole other set of, um, let's say silo busting that they're being asked to go do. So, um, scope changes, uh, as well, including not only security, but a lot of questions we hear about legal risk and policy and understanding of what the legal implication of the models are. IP protection of the model data usage.

So it's, uh, kind of an expanded scope, both in terms of breadth and depth of what people are being asked. At least we start, we're starting to see that kind of bubble up in the, at least in our customers. Yeah, I, I think those are all really great things that I, I'm glad you guys are paying attention to.

I, I get inundated with questions about trust. Um, and people largely come to track marks and ask, how can we trust the AI generated code? Um, how can we be assured that the AI remediation is actually going to fix my vulnerability?

Um, they come to us with questions about, you know, how do we put in bend but don't break policies where we can, you know, give the developers an inch and say, yes, use these new tools, but also come back to our higher ups and to our board and say, but here's how we're evaluating that code. Here's how we're scanning that code to make sure that we're not introducing more risk into our pipelines. Um, and so we work with them on scanning that code in real time.

We work with them on understanding how you can use reflection in a model, right? So we've come up with this idea of a confidence score, right? When a model gives you a response on how to fix something, you gotta question that model.

How confident model are you that the code you're giving me can actually go remediate my vulnerability and have it explain how it came up with that solution? Because education is, is a continuous thing, or at least it should be, especially this day and time with all these new technologies. So if developers are going to use these tools and apply 'em into our code, we have to give them a chance for that human intervention that, uh, we always hear the human in the loop.

But I think that's one of the most important things is people come to us and say, how do I trust this? How do we put in policies in place that are gonna allow us to be able to sleep at night, uh, when we're introducing this new code into the organization? Great.

Had Tyler, speaking of trust that Steve brings up, one of the conversations we don't have yet is zero trust. And how should we treat software that's generated or created through ai? O one of the questions I asked, like testing companies early on is, do you think of code created by AI as a junior developer?

Do you treat it as any other developer? Do you treat it as the most suspect of, we don't know where this came up from. Maybe it's like an open source project we've never heard of.

What, how do we treat, do we need to apply some, some zero trust type principles or we have to take a totally different approach when we think about AI and development? I think that academically, I could sit up here and talk to you about zero trust and as a, as a, as a framework. And, uh, I'm not gonna say it's bad, right?

But I also think we all work and support for organizations that have goals of, of completing a mission. And in that case, it's this kind of bend don't break set of policies we see the most. And that iteration, and it's been said before, but I don't really have a better metaphor than the early days of open source and that kind of, um, iterative disclosure and trusting it more.

We don't see, it's not necessarily a, a good set of policies, a good set of tooling, a good set of processes can work and I think be effective regardless of what or who writes the actual code. But what we don't see is any removal of accountability in that RACI matrix. What what we don't see is any removing of the developer that does the merge request or the pull request still carries that accountability, uh, and responsibility to the work that they are, are, are doing.

And that I think is the foundational, uh, thing of success is that accountability is never removed. There is that pull request, that merge request who somebody's name is next to somebody's job. It is to make that happen.

And that, uh, think sets a foundation of process and policy and trust, uh, that we see kind of as a, a good place to start. Mitch, we might be looking at this the wrong way too. I think we're sitting here thinking, is there another security control that we need for ai?

And I'm gonna just sit here. I think the security controls that we have today are pretty decent, like ing the bigger risks. Like I'm worried that my developers are putting in intellectual property into these models, right?

What controls do we have out there guys that stop developers from doing that? We have DLP, right? These are security controls that we should already have in our organizations that should technically work for some of these risks that we have today.

What we're most concerned about, I think, and we haven't really voiced it yet, is the non-deterministic factor of ai. Like in computer science, we know this, you have code, you build it, and you end up with a binary, right? And you build that thing every single time the same way you're gonna end up with the same exact binary.

You do a hash on the thing that hash is gonna come back the same exact way, right? You know this, it's deterministic. The only problem is with ai, everything's changing all at once.

It's just so much faster. And you can't run something through a model on day one and have that exact same thing happen on day two or day three or day 10, because that model just keeps changing if you do it that way. So we're trying to explain it to ourselves.

Are our security controls good enough? I think it is for now, but we still need to understand how our developers are using ai. And if so, how is that changing our day to day?

And if we can't figure that out, we're always gonna be behind. And I always think security is a laggard behind technology. Technology's gonna happen with or without us.

We better play along and play. Nice. Alright.

Look, I, in principle, I agree that existing practices are, is a very good start. However, I also think we should keep in mind the fact that there are some attack vectors that are unique to ai. My favorite example is all these code generating models, uh, always hallucinate, uh, dependencies.

Mm-hmm. They make up open source dependencies that don't exist. And some of them do it in a consistent way, meaning they would hallucinate the same dependency over and over again, which kind of gives you an opening to create this dependency in real life and, and do it maliciously.

So that's something a human developer doesn't do. Okay. Just as an example.

So I think yes, existing tools and practices super important and apply to the code being generated by the agent, but there are a few additional things to also keep in mind and take care of when you let, uh, when you let AI write your code, It's kinda like typo squatting for open source, right? Yeah, exactly. Packages and libraries.

Exactly. Steve, any thoughts on, on the trust side of you or the one that brought up trust? Yeah, no, it, these are all really great points.

Uh, I, I think that there are a lot of good measures in place. Um, ultimately I, I feel like this is more being driven by engineering than anything else, right? At least in the early days where we're seeing a lot of adoption for ai, it's with developers either using it to write code or building applications where they're embedding ai.

So for me, on the trust side, uh, it's important. Now we always talk about shift left. I think we've sick of hearing that term.

Uh, but now more than ever, AppSec should be sitting with engineering and understanding what your developers are doing. How are they using ai? What types of applications are they trying to build?

You need to understand the roadmap. You need to understand if they have plans to adopt these libraries. You need to start putting in and understanding and building policies internally where you are comfortable with AI usage, right?

Maybe we say, okay, yes, it makes sense. You're gonna use you, uh, AI on the, the front end. Cool.

If we want you using it for authorization, maybe not right now. Why? Because trust takes time, right?

We're not just gonna trust everybody overnight. But that's how you can start putting systems in place where your organization can monitor the usage, understand the usage, and then build practical policies around it, right? So to me it's, it's crawl, walk, run, and, and you know, don't throw the baby out with the bath water, uh, to, to your, your points guys.

And then there's been a lot of great things that we're already doing. We have to keep doing them. Um, but you know, I think just really sitting with engineering and understanding where their push is coming from, where they're trying to use this technology is, is vitally important.

Okay, this next question's got a little bit of wind up, so hang up, hang on with me for it. So we've gone through this transition of thinking about security and software where rather than deploy software stay stable, we don't change it. That's how we know it's secure because we don't deploy software very often to an, an environment that we deploy software quite frequently, maybe multiple times a day, multiple times a week.

I like to use the analogy of software's not a solid, it's a fluid. It's constantly changing, whether it's open source or third party or APIs or SaaS services, your own code. So that's the windup to your, to your point Naomi, I, I wanna explore that a little bit further.

The non-deterministic nature of generative AI doesn't mean it's gonna generate the same code the same way every time. You may say that fixed and solve it, try it again. Who knows what changes.

Mm-hmm. Are we in a better spot today because we have automated processes today that handle changes in code that we aren't predicting coming through the development pipelines. Is that gonna help us address some of what you brought up?

I mean, if I wanna be honest, the processes and tools we have in place are not sufficient for code that changes all the time. But if you're still building code, writing code, building code and running it somewhere, hopefully your controls are good enough for that. I'm worried about more of that age future that we all talked about and listen to for the past few hours.

That part scares me a little bit. I don't even know what that future looks like. Honestly, More philosophically, I think potentially there could be a lot of value to the lack of predictability of these models, right?

Because from a security standpoint, when you are too predictable, it makes you open for an attack. Whereas I don't like, I don't have anything concrete. I'm just thinking about out loud, out loud.

Maybe you can find a way to leverage the non-deterministic nature of AI to create more security rather than less Subcu security through OBS security. Yeah, exactly. 'cause it's not the same every time, right?

Interesting. That any thoughts? I don't, I'd hate to make a judgment call on better or, or worse.

What I think is we're in a, uh, state of I think iteration that some things are better. I, I know plenty of, uh, engineering friends that like would rather speak and let the code be written, move up the level of abstraction. I think that in that way it can be very good.

Um, and I think that just like anything from a security perspective, I think there are some aspects of what we do that are better now. Um, patching speed and, and, and frequency. Um, so on net I feel like we're raising the, raising the floor.

Um, and that comes in both good and bad. Uh, but uh, what's that Chinese, what's that proverb? May you live in interesting times.

It's interesting, yeah. Getting more interesting by the moment. That's Right.

I I think one of the ways that we, uh, can also address this is by, um, not making it so unpredictable. I mean the, you know, part of what the big effort is, is building large language models, models where the output is more predictable, right? And we can do that.

We can red team, we can ab test, you know, we can verify and, and pre-check just like we do a lot of things that one plus one is still equals two in that model. And then as soon as we start seeing deviation from it, 'cause we're doing things like continuous testing, then we know we have an issue, right? But more than anything, right, when we think about the impact of, of ai, the, the amount of new code, how fast everything is, is being developed.

If you don't have good CICD processes today where you're a automating builds, running union tests, doing uh, you know, uh, automated testing, this is, this is gonna be extremely painful for you. 'cause you're gonna need those good checks and balances. You're gonna need to be able to make sure that you've got the good regression testing in your models themselves, uh, to, to really I think get to the point where you're talking about of of having that trust.

Great. Jake, is that you out there? You have a question?

Um, so, you know, we've sort of talked about one side of AI here, which is how you empower development with ai, but the other folks who are being empowered by AI are the bad guys. Absolutely. Um, and I think one of the things that we're gonna see, and we saw, um, some of this in the Mandiant and the, uh, DBIR report last week is there's gonna be an increase in zero days, um, because it's so much easier for folks to come up with really difficult attacks or sophisticated attacks.

So how do you see the future of defense against zero days based on the fact that it's gonna get easier and easier and easier? Yeah, I mean the, the really scary truth is, um, if you think about how far we've come from a, let's say publicly accessible productization, you know, the companies we represent, what we're building out there, um, our attackers are years ahead of us right now, right? So they have a large runway to work with and it's gonna take the AppSec market, I think years to catch up until we can balance things out and be playing on a level, level playing field.

So, um, vigilance is, is gonna be key. You know, we've got a lot of research at check marks that, uh, kind of to your point shows that already attackers are doing things like, um, open source, large language models and poisoning them, right? We were able to prove remote code executions.

We're able to show that these models are now really the, the kind of vehicles for malicious packages. So your spyware, your ransomware, your malware. Um, and so, you know, when we think about zero days and we think, you know, for for us, it's, it's gonna take us, I think all collectively as an industry to stay on top of this and, um, to really look out and start thinking about creative solutions and education around what the latest threats are and building them into our threat landscapes.

Uh, but I think it's, it's a, you bring up a very, very good, very big challenge in the market that, that we're all facing right now is how we're gonna close that gap. If I was, if I was a threat actor and I'm, I, I'm not, but if I was, I would go into GitHub, I would point my mo my model at it, I'd be like, tell me the top 10 most popular projects on UB and what vulnerabilities exist in those projects. I would then try to go to a SAANA type.

I know we have a saana type here. I would then see where those projects are actually used. Like if it's an open source library that's popular log for shell log for J is a popular one.

For an example, I would then ask Sonatype point my model to Sonatype. How many open source projects use that vulnerable library or have that vulnerable thing that I'm looking for, right? How many of those are in commercial or uh, cuts software?

How many things are sold? How many things are for free, right? Like, and then I would attack that all day every single day.

I think that's, it's like log for shell, but more all the time. Yeah. And I don't wanna like scare anyone, but if you don't have the visibility into your application since you know what's actually running in your applications, like how are you supposed to defend against that?

Right? And I will say there's a pretty good software out there. Contrast does that pretty darn good.

So I'm gonna say like, if you guys don't have that visibility in your applications, check out contrast pretty great. Alright. Look, I'd say fight fire with fire, meaning I think we're already starting to see where agents are being used for protection and for creating better and faster security to combat those, uh, bad actors using AI to create the attacks.

So this is something we are working on, but I see many other people do it as well, which is deploy agents to on your side in your defense, that can act as quickly as the ones against you. Yeah, I would say, I was gonna say the same thing. Fight fire with fire.

Um, 'cause we're doing similar things. I would also say that there's the, uh, rapid response that can further be accelerated by the AI agents, acceleration, tech technologies. Let's say you're not perfect.

Uh, somebody breaks through the offensive line to go into the, to the secondary. It's how fast you can collapse on that. So it's visibility agents help with that.

So you can fight fire with fire, both in protection and then rapid response to close that. I think that's what we're seeing is, um, there's a lot of hardworking, talented people on trying to protect, um, and innovate in a lot of those areas. So I think that's, that's exciting to see.

Great. As Mark comes up, hold for a moment, please thank our sponsors and thank all of our sponsors that were part of, uh, helping you bring this to us today. So I'm sorry we didn't get to your question, but we'll try to talk to you in the back.

Okay. So I hope we'll see you again next year. Okay.

Thank you. Thank.