AI Is Moving Faster Than Security

Hey, everyone. Welcome back here to Tech Drunk tv. You know, as we go into the new year, I couldn't think of a better way to kick it off than with my friend James Wickett.

com, like in 20 13, 20 14. Of course, uh, especially as we got into the whole DevSecOps thing, James was one of the, it was one of the early kind of prophets in the wilderness, if you will, with rugged DevOps and all of this stuff. And I always knew he was destined to have his own company, that he had that passion to go found something and build it.

And so I was thrilled when he started Drive Run Security, but I, I don't want to tell James whole story. James has to tell his story. James, welcome to Techstrong tv, man.

It's good to see you. Yeah, yeah. Thanks, Alan.

I, I appreciate it. It is, it's, uh, it's been a, it's been a fun journey, hasn't it? Like, you know, kind through Yes, it has the Devon DevSecOps, the DevOps era, kind of just all that stuff we've been been working on, and, and you, you've really done a great job kind of building the community, helping people understand what's, what's going on, uh, with boots on the ground, uh, in the world.

So we always appreciate, um, your, your coverage. Appreciate you, man, your leadership There. Yeah.

Thank you. Yeah. I appreciate it.

Yeah. Talk to us. It's, and, uh, yeah, yeah, sure.

Okay. So look, what's a big deal, right? Everything's AI these days, right?

And everybody's talking about it, uh, six ways, uh, differently. And, and, and I think like, there is, there is an issue with, uh, AI is like, we're building AI applications in all the systems and all the, uh, uh, in all, all the workflows we have in, in any enterprise, in any organization that we have today. And the the issue with that is that, um, we don't really know, uh, the risks that we're facing, right?

Like we're, we're taking sort of like an untrusted compute model, and we're putting that into, into our system. So, um, but, but to back up on, on that, like, uh, Alan, I started the company because I felt like, yeah, we did the shift left thing, but like, man Shift left really never delivered anything for developers that was as tangible as I think that we wanted to, right? Like, we saw incremental gains, but we never saw, like, you know, I, I kind of, I looked up and I, I knew like developers weren't having a good time.

And I talked to my co-founder Ken, and I was like, we gotta, we gotta start something. Like, we gotta help developers have a good time with security. And a lot of it is because we took a lot of tools as an industry, I'm just kinda using a, a stereotype here, but as an industry, we took a lot of tools, uh, that were made for like a different era, a different user, uh, and kind of for forced them on developers, uh, sort of like retrofitted that.

And so, um, that, yeah, that's, that was kinda like the genesis of like why we started Dry run security. 'cause we wanted it to be like, you just dry run your code and security just sort of happens for you. And like, and, and it, and it makes it easy.

Uh, I don't know if you remember when we first started, we were always talking about the security buddy. Like, you know, we mm-hmm. We didn't have the language of agents.

We didn't, you know, but even from the very beginning, we were like, yeah, we're gonna build, we're gonna build a security buddy, uh, for developers, and it's gonna be like right there helping them, helping them make this happen. Um, and so, uh, the companies, uh, we're, we're now in our, we're, we will be in entering in our third year in January, uh, kind of moved out of, out of stealth and into, uh, shipping, um, uh, products for, for customers. And like, um, right now, I think we, this, this month we'll have done 250,000 code reviews then this month alone, uh, for our customers.

And so I remember when we were happy, we did like three in a week or 10 in a week, you know? No, no. It, that you know's called Scales my friend.

Right. Good for you. Yeah, it's been growing.

Yeah. You know, you know, James, in hindsight, so I'm gonna ask you to answer this truthfully. Okay?

Yeah. In hindsight, in hindsight, it seems like dry run security, its whole mission, and the idea behind it is like tailor made right? For AI and Agentic ai.

Were you thinking AI when you guys first started talking about this three, four years ago? No, no. We, we launched in, um, or we, we kind of went to go fundraise in 2022.

And the thing that we were working on at that point was, um, how to have better sec like security that didn't like waste everybody's time. And so we were trying to make some connection points between DAST and sast. And the, the reality is we were really hunting for like, what's really exploitable and what's not.

I mean, I know that we've had a lot of talk about reachability, but we were really on the hunt for exploitability. Um, and so that's, that's what we did our, in fact, our first six months, we, um, we built some really cool stuff, uh, that later we've more or less kind of trashed, you know, because we had people try it. And a, it was really incredible to get people to try it.

Like, I couldn't even get my friends to take the five to 10 minutes to try it. They were like, oh, yeah, yeah, sure, next week. Okay, I'll be free next week.

Right? And so, um, It's always next week. Yeah.

And so we can, but at the same time, while we were building that, we were also, uh, doing a lot of experimentation with ai because we saw, even from those early days that AI would be something that could help us deliver, uh, you know, our documentation. We felt like there could be that chat buddy that could work out there, there could be some other components that we could, could put in place. But as we experimented with more and more, uh, we really built AI from the core ground up.

So right now, um, when we talk with customers, we talk about our four key agents that we ship. We ship our PR code review agent. We ship, we ship, um, a, a custom policy agent.

We have an insights agent, and then we have a, a new agent that's gonna be launching, uh, in January that we'll, we'll be talking about coming soon. Uh, and, and, uh, you know, spoiler alert, it's gonna be looking across, you know, all the code bases and kind of do some really interesting stuff for folks. Very cool.

Yeah. Very cool. We'll, we'll, we'll be on the lookout for that.

Yeah. James, before we go further for people wanna get more information on dry Run security, what's the website? Yeah, it's just dry run security.

Um, and, and, uh, our, the, the kind of the, the guide, one of the reasons, like we were, we were talking about, um, what would be good for, I think for your audience and your listeners is like we, uh, we wrote Building Secure AI applications, which is a 40 page guide, uh, with a reference architecture that's in depth controls on it. Yeah. Yeah.

We, we kind of were like, we wanted to put a lot of bones on, or a lot of meat on the bones for like the o os top 10. And so like the O os top 10 for LLM applications, which is radically different from the OO top 10 for, you know, regular web applications. Yeah.

You know, before we jump into that though, James, I feel like, you know, everyone talks about AI today, as you said earlier, six ways from Sunday. It's all people talk about, but we we're, we're, we're really crappy on definitions. Do you know, so what, what is an AI application to you?

Okay, so what, What define an AI application? Yeah. When, when we think of AI applications, and whenever we're talking about, uh, people using, um, l os inside of their product.

So this can be, um, small things of like, um, I'm putting a, a new chat bot inside of my, um, inside of my website to interact with, with users, and I'm providing this, uh, with that, and I'm hooking in some backend data for that. Um, it could be, uh, internal, uh, applications. It can be MCP servers.

Uh, it can be ways you're, you're leveraging, um, uh, LMS or S SLMs to inside of your, your applications. And so it, we kind of think of it out broad scope, but it's a, uh, and I, and I know that, you know, you just asked me to, to define it, and I give you another kind of broad definition, but it is like anywhere in organizations where, uh, they're leveraging and putting LLMs into production for, you know, usage with either internal data or ex external, uh, facing stuff. Excellent.

So, yeah, I, you know what, that's as good as any 'cause I, I think one of the problems we have in with AI these days is just agreeing on basic definitions, right? Yeah. And, and so I think that's a good way of looking at it.

Um, I think it's, uh, Alan, for, for me, it's like it, whenever you put an LLM in, you're, you're changing the risk model for your application. Um, because, because now you've given it access. You're, you're, you and, and you've given access to different data types.

You've given it, um, uh, access to do, uh, some probabilistic decisioning, um, on your, on your system. And, um, totally. Okay.

I think that, I think I'm very bullish on, on that. I think that's really great. But, uh, it does change, like what, um, what you have to look for from like a risk perspective.

You know, for me, Jane, so this before my time, but you've probably heard these stories too, right? When the telephone first came out, it wasn't like person to person. Yeah.

It was kind of a party line where, uh, you know, anyone could listen it. Yeah. I think when you bring an LLM in, I mean, the good part of it's that you have access to the entire corpus of that ll m's information Scope.

Yeah. Scope, yeah. Yeah.

The bad part of it is everything you give give it, it basically ingest as well if you're not careful about it. Right. You know, unless you take precautions and, and so you're getting this information from everywhere and you know, the old saying crap in, crap out.

Right? You don't know where that information necessarily came from. Yeah.

And you don't know what, what's going in there. So, and so the, it's almost like mirror image of, of security issues around these AI applications. Right.

And, and I, I assume in the guide, you, you've gotta address both of those. That's right. Yeah.

Yeah. You can, we, uh, have some things like, uh, we, we talk about excessive agency. Uh, we talked about, uh, the different types of guardrails and policies after you put in place, uh, for that.

Um, we, there, there is, yeah. It's, it is, it is very different. And like how you, how you think about what, what data it has access to is, is like, gotta be, you know, forefront, forefront of mind.

And, um, I, the thing that, that's interesting to me, Alan, is like, um, we haven't, we don't have conversations where anybody's not using it, like everyone's experimenting with it to, to some degree or another. Um, and then there's, there's a bit of a, a hurry up on like, well, well, kind of what kind of controls or how do we safeguard this? Or whatever.

And, and that, that conversation, um, I think that's gonna be the, one of the main conversations we have in the, the next year, two years, uh, is, is around that, you know? So, But, but you know what, James, that's not a new conversation, conversation for security, is it? Right?

No, it's always, you know, you guys, hey, the train's pulled outta the station, hop on this moving train and make it better or make it secure in this case. Yep. But, you know, but the train's already moving and, and so, you know, we can't dig in our heels and say, Hey, go slow or, or be careful.

No, people are going as fast as they can, it seems, because there's such pressure to AI eyes, ai, everything, you know? Yeah, yeah. Um, James, well, And, and there's that discovery of value business needs the value.

Yeah. And so it's like, Hey, look, we, we see the a path to value through through ai. So that is a, uh, that that is gonna be a feature where security has to come alongside and, and enable that, right?

So, yeah. But here's the funny thing. Yeah.

I don't know if we've found the killer app for AI yet, Right? Yeah. Well, certainly we're, I mean, Dry run security is certainly the killer app for, Okay, there you go.

You stepped up there. That was a softball, James. Yeah.

Yeah. But, but you know, seriously, like we look at development, what you, what you said there is a hundred percent true. I've seen, I've seen, uh, studies, 90% of developers are using AI and helping them develop code, right?

90%. Yeah. 40% of those 90, almost half.

Right. Don't trust it. Don't trust it.

For sure. 65% of those 90% think it introduces instabilities into their code base, but still 90% use it. There's something almost illogical about that.

I, I mean, what's the definition of insanity, right? I I use it even though I know, I, even though I don't trust it, even though I pretty sure it, it introduces instabilities, I'm going to use it anyway. Why?

Because everyone's using it and they tell us that's what we should do at, at some level. What a disconnect. Well, I, I think they, they are also seeing the value of like, being able to go faster because like, they, like some, some engineers and, and it depends on the engineer, and I think that depends on the company and certainly depends on, um, which, uh, you know, which coding assistant tools you're using in your stack there.

So I think that there, there is some, some disclaimers around all that, but yeah, I think like the, there, there are enough benefits of like how much code, uh, velocity, like we're able to see. Mm-hmm. Um, you know, I can't, you know, we, we've been, uh, doing a lot of customer adding throughout the year, but like, and I mentioned we're at like 250,000 code reviews a month, right?

But like, even within like a certain customer like slice, um, like we've seen them continue to grow faster with the same or marginal amount of, uh, developers. Now, I, I don't have like a pre-baked number for you, but I wouldn't be Surprised. One and half for Two x or so.

Yeah, Yeah. No, I've seen numbers. Yeah.

The story supposedly is we're four X-ing the amount of code we generate four x That's crazy. Crazy. Four x.

Yeah. I don't know if we're four X-ing the security of that code. No.

Yeah. And that really is, is the crux of it, isn't it? Yeah.

Yeah. Well, and, and the, the, the real issue too is that a lot of these tools, um, from the last generation of pattern matching tools, they, um, they weren't really doing a good job before. Um, and, and not, not to, to discredit them.

They had the tool that there was, there, they had, you know, regular expressions and be able to do matching and stuff like that, right? Nor normal, normal, um, opposite of stuff. But, um, it, they, they underperformed in, um, abilities where they had to find logic problems, uh, authorization issues, stuff that like really took like human coder viewers.

And that's why we spent all the money on bug bounties through the industry. I'm talking at large here, spent all the money at bug bounties. We spent all the money on like, uh, human coder viewers doing that.

Um, I'll tell you a funny story, Alan, uh, uh, let's see, about four months ago, uh, showed up to a customer. We ran a free, uh, free complimentary scan against their code base, and we handed them the results, um, and they said, yeah, just email it. And then, and then, uh, we, we will, we'll meet on Monday.

And so that was on a Friday. Well, on Mon on Monday, when, when he showed up, the customer was like a little bit incredulous, uh, uh, felt like he was a little bit angry with us. Um, and we were kind of like surprised by that.

And I was like, Hey, what's, what's the, what's the deal? And he's like, what? I looked through this and you found 20 real issues out of the 20, the 20 issues you reported, all 20 were real.

I filed bug tickets. I went through every single one of 'em over the weekend. And, uh, and I've been using this, I won't name it, but have you been using this other product for six years?

Six years in the same code base? They've never told me any of this. This is all like net new information for me.

And so, um, I think like we have to realize like there's some really positive benefits we can get out of all that. So of course, and this, We've also gotta realize that ignorance is bliss. Sometimes There is not.

There Is that, yeah. Yeah. It's like, Hey, there is new work, but, but wouldn't you rather, I, I, I feel like, and maybe I'm too much of a purist, but I would rather know about that earlier.

Um, and I wanna shift that left and then, and instead of paying the penalty of the bug bounties, the No, I, I, I do, I do think to a certain degree, ignorance is bliss when it comes to these things. Yeah. No harm, no foul.

Because, you know, and this goes through the whole, as you know, I've been in security a long time. I think there are a lot of people, very content to be just another zebra in the herd and hope that the lion doesn't pick them on any given day. Right.

And, and so Yeah, I know I got some lost top 10 issues. I know we've got some stuff, we'll get to it. Yeah.

But, you know, I don't have the resources. I don't have the know-how, I don't have this, that, or the other thing, why I could fix all these things now. And I'm not banging anyone, but it, it's almost the nature of the beast in security, right?

Because we're so used to just, we do the best we can. Yeah. Right.

It's not perfect. Yeah. And, you know, there is that mentality that I think we have to overcome.

James, I want to turn back to this guide. Yeah. Yeah.

40 pages chock full chalk full of, of best practices, emerging practices on building and securing your AI applications. Yeah. If I had to say, James, what are the three biggest things people should take out of this?

Or they definitely should go check this out. Yeah. What do you think they were?

Okay. I think that, yeah, that's, uh, I think that, um, number one, we put reference architectures in there. So we put, um, we've taken a couple slices of what a, uh, AI application looks like, and then we put in like the type of controls, uh, you can affect both at a runtime and a code level.

So I think like, just getting it for the reference architectures, that's, that's really good. And, and it's all free. You can get it on our website if you want to download like a portable, like a PDF or something, um, you can do that.

But, uh, it's all just like available, uh, publicly ungated on, on our website. Um, but so, so, so one, one is the reference architectures. They're incredible.

Two, we, um, we've kind of said for each of the, uh, oasp, uh, uh, areas, uh, functional areas, like something like excessive agency or prompt injection, we suggest to vendors. Uh, like that you can look at both on the, on the runtime and on the, uh, on the code side, uh, to help do prevention or control or things we think that are interesting there. Um, so if you're kind of in like a, you're in a shopping kind of mood or you have a problem with this kind of thing like that, that is a real helpful, uh, guide, uh, for that it helps you isolate, uh, which ones would be good.

Yes. Dry run is in there. We're of course, like we're, we're a vendor, but like other companies are in there as well.

It's not just like a buy dry run and solve all your problems. It's, it's far from that, right? It's just like we're, uh, sure.

Yeah. We're one, one piece of that pie there. And then I, I, the one thing that I like, uh, the most about the guide is that we also tell stories like concrete, uh, examples, either from the news or from our own, uh, our own, uh, experience.

And so, um, one of the things that, that's a funny one for us is, uh, uh, I think it was in July, Alan, we spent, uh, maybe two grand, uh, $1,800 and about a 24 hour period for one customer who wrote one policy that went a little bit haywire, and it was running nonstop and doing that. So, um, whenever people commit code, we have our custom policies come in and they're able to look for, for problems. Um, and then they also have access to go back into the code base and find, and I'm sorry, excuse me.

They would have, they're able to go back in the code base and find, uh, other code that might be related to the problem that you've described. Well, this, uh, policy didn't have enough guardrails internally on that, and it started looking across the whole code base and trying to like, do a bunch of analysis and it just wouldn't stop kind of pulling in GitHub. And this is one for one of our bigger customers.

And as they continue to write, uh, more code, I think they probably shipped 50 or a hundred port requests during that, that day or that that time period. Um, our policies, uh, you know, it was just, it's just threading out and it's just like trying to, you know, look at, look at all the code and do a ton of analysis for it. So it was basically running hot for, you know, 24, 30, 30 hours.

So, so, you know, we have our own, our own internal scars. You know, we're building, uh, AI systems just like everybody else. And so, um, you know, we're, we're a small startup, but I could imagine that happening to like a, a big, uh, you know, you know, fortune 100, fortune 50 company, and, and that being, uh, you know, a, a half a million dollar hiccup or a million dollar hiccup, right?

So, um, but even for us, like, for us to spend that amount of money in, in that window is, was, uh, was crazy. But that, that's the kind of stuff is like, we're, we're trying to add in, uh, uh, we as the organ, as the, the global we, um, we're trying to add in LLMs and build AI applications in our systems, and it's just, it's gonna impact this in ways we, we didn't really see before the type matches for what we've had before, right? It's like, yeah, we've pegged all of our CPUs or we've blown out memory, or we've, or the connection pool's expended.

Like we've, we already have that, that model of, of working in, in, uh, in, uh, our computer science lives. But yeah. Um, yeah, this is just a new way for us to kind of see some of the same patterns, uh, arise that we've had before.

So it begs the question, right? So I think back, James, the first company I started Tristar Web, Okay, 95, 96, we became what became a hosting. We didn't call it hosting, but eventually it would became known as web hosting.

Yeah. And, you know, I used to monitor the, so I was the overnight, you know, it was your company, you gotta, you're the chief, you know this now, right? Yeah.

You're chief cook and bottle washer. So I would go overnight and monitor things and, and those are the things I used to have to watch my CPU usage, my disc usages, the bear, you know, all of the monitoring kind of dashboard. Yeah.

You think we'll have a, an AI security dashboard, is that something drive, run? Yeah. Maybe has in its future?

Yeah, I think we, we kind of stop at the build the build time. So we're really concerned with, uh, we do provide, uh, code level insights across like all the ways you're using, uh, ai, the, all the MCP, uh, uh, services or, or that you might be, uh, discovering. Um, but yeah, I think that there is like, uh, kind of a host of like, uh, like we're seeing a rise Phoenix and some, uh, LLM uh, operations, uh, dashboards spin up.

And so yeah, I think we will have more of an ai, um, ops, I, I guess, yeah, we'll have ai, ai ops, Not, not AI ops that we used to use that word, a new, a new version, A new AI ops. Yeah. You know, yeah.

Words are hard, right? But, but I, but I think like this ai, the, the, an AI SOC too, but the mm-hmm. But, but not just, I think a lot of the AI soc companies now are mostly concerned with like just replacing the, the SOC function with AI function or augmenting that.

But I think to your point is like we, um, we'll still need some sort of like, what are all the AI agents doing, right? And then, and then understanding like their, both their access and like keeping in, keep an eye on it. So, um, oh, you know, I think one of the things that's helpful for me, and it's always a frame frame is like people ask, well, why is this so different?

I'm like, well, you have, you had developers, uh, writing code and that's a probabilistic system. And, and, and so like a deterministic system, like a pattern matching tool just was, was always struggling to keep up. Now we have like a probabilistic system developer using another probabilistic system, uh, ai.

Mm-hmm. And, and so, um, that is not going to result in like a better outcome for, you know, the, the, the pattern matching approaches, right? You actually need another, It's not like a negative number and a negative number equal a positive number.

Yep. Yeah. You actually need a, a l LM based AI security system looking for that.

And we use internally even we have like our exploitability thing that I mentioned at the top is like our, uh, you know, we have a way to judge, like we use LLMs as judges of the other LLMs performances and like, so the agents, you know, kind of will put that pressure on each other. Very cool. Yeah.

James, I'm assuming the guide is available to anyone who goes to drive run security. They could download it. Yep, Yep, yep.

Dry run security, there's a big, uh, there's, it's either under our resources tab or we have a banner. Uh, and we'll have that banner up for, for several, uh, several weeks here. Um, and, uh, probably even through RSA, um, and can go check that out.

But we'll have that. Very, Very cool. We we're doing, you know, this year far, I say we usually do the DevSecOps thing on Monday.

Uh, this year we're doing it on AI native dev or securing AI native dev. Yeah. And we've got like Patrick and uh, guy Ani from that community coming down to talk and stuff.

It's, it's gonna be interesting. That's great. Looking forward to it.

Hopefully I'll see you out there. Okay. I'll Be there.

I'll be there. Yeah. Well you, I count on seeing you.

It's gonna be fun stuff. I'll talk to you about it. Anyway, James, we're about outta time.

I want to thank you for coming on here. Continued success. Keep doing what you're doing.

A dry run. It's, it's exciting. For those of you watching this go to Dry Run Security, download this guide on building secure AI applications.

Can't hurt James. Happy New year to you and the family and I will, uh, speak to you soon. Speak to you soon.

Thanks Alan for having me. Alright, James Wick it, CEO co-founder Dry Run Security here talking about their new guide on building secure AI applications. We're gonna take a break on Textron.

We'll be back.