Why Enterprises Remain Cautious About Using AI Coding Tools in Production

Hey guys, thanks to the throw we're here with Nigel Douglas, who's head of developer relations for Cloud Smith, and we're having a little chat about, well, just how secure is all this code that we're creating with these AI tools, because well turns out there are a lot of issues to think through here. Nigel, welcome Michelle. Thank you very much for having me.

I think everybody at this point who writes code has at least experimented with an AI coding tool if they're not using it every day. But I get the sense that not everybody kinda understands like just how insecure these things are. For example, you know, an indirect, uh, prompt injection attack can be pretty lethal.

Um, what's your assessment of these tools right now from a security perspective and, and what should developers be looking out for? Yeah, that's a few things. I, I think the main thing we need to remember about all these systems when we're talking about large language models, is that their probability statistics models at the end of the day.

So it's really about predicting the next word. Um, when we think about that nature and the fact that it's non-deterministic, it means that if we ask the same prompt 10 times, there's a chance nine people get more or less the same answer and one person gets a different answer. And that different answer is really between, am I pulling a software package or using code that doesn't make sense?

It's not real, it's fabricated versus nine outta 10 times. It's something that genuinely exists in a code base and is reproducible. And it also seems though that the bad folks out there that are trying to compromise our software supply chains are getting savvier about how to fool those LLMs into doing things that we don't want 'em to do.

So, um, our, as the software supply chain as we currently know, it gonna be much more easily compromised A hundred percent. You know, that, that's a point adversaries will always look at. So if we think again about it being a probability system that's trained on existing data, we have to think, where's that data coming from, you know, the likes of GitHub or Reddit.

And if I were an adversary, uh, what I would be looking at is, again, in the case of type of squatting, it, does a package exist that sounds like a plausible package that someone's gonna ask for? And if it doesn't, what if I started fabricating these packages and creating and uploading these packages in public upstreams on the basis that eventually LLM might say, well, here's a suggestion. Why not try that package that I know exists somewhere?

But of course it's not good. It's, it's full of malicious code. Um, so that's a thing that's expedited by LLMs is this concept of slop squatting.

This idea that, look, if we rely blindly on the output of AI slop, you know, we don't know what we're taking, we're just asking and applying. Um, you may be susceptible to pulling something that an adversary is put there deliberately in the hope that you consume it without thinking. So yeah, it, I think LLMs it's, it's a fact at this point are expediting risk in a few different ways.

The deliberate element where adversaries create malicious patches on the hope that someone accidentally consumes it via LLM. Um, but then there's the other outside of it is, which is, you know, when I started working with Kubernetes, I learn, I take samples from the, the docs and I apply and I start making changes based on deliberate decisions, which is maybe I want to only give access first specific scope, you know, from a network policy perspective, maybe I only allow ingress traffic and know egress out of it. Uh, and these kind of decisions, um, when it comes to ai, if you just say, create me a workload, it doesn't immediately say, well, I'm gonna create the workload with these security considerations in place.

Because unless you prompt it and ask for that consideration, people are blindly consuming code that is generated that is arguably insecure. And that's not deliberately to be insecure. It's just, it wasn't part of the prompt consideration.

So yeah, there are so many things to think about from a security perspective that I question our developers primarily focused on the security aspect. I would say probably not. Mm-hmm.

Are we just waiting for some sort of cataclysmic event then before we all wake up and smell the coffee? Because right now it seems like, you know, everybody and his brother's so obsessed with the productivity gains that nobody wants to talk about. Anything that might be, well, shall we say a downer?

Yeah, I, I, I think we may have had that incident in the last couple of days. So, you know, there's one element when we talk about the likes of the Claudes and cursors, these tools that already exist, but people keep pushing towards, can I have it do everything for me? This fully idea of agentic, you know, if I could ask an AI agent go and do something and just run as a background task, will it make me more productive if I'm just on autopilot, I'm not involved.

And you, you may have heard the open claw projects that came out and well, it's grown in popularity in the past two weeks, but essentially what you were doing was running this on your local hardware and saying, here's route permissions to go and be agentic and do what you need. And of course a lot of people are compromised in that time. You know, people are pulling skills from public repositories 'cause it's all done for me, this is fantastic.

And of course, we don't know what's bundled in those skills. We don't know to what extent this agent will operate, you know, will it accidentally delete something or will it deliberately delete something? And what impact will that have on us as a organization?

So yeah, these are the sort of incidents that are happening now is this blind trust of, well if it's ai it's gonna be better than me, but what adversaries compromise that? So yeah, that, that's probably the first real example we've seen so far. And we don't even know what other AI agents, those AI agents might be hanging out with.

'cause my AI agent could be perfectly benign, but those other ones could be fairly malicious, right? Yep, absolutely. So the idea here with this open claw is fundamentally is open source so anyone's free to contribute.

So the idea of any type of hub since the beginning of time, if we think GitHub or we think of Pythons, pi, pipi or NPM for JavaScript, all of these are public repositories where it's fundamentally the core control of open source is that anyone should be allowed to contribute. But if I'm bad and you are good and you're going to your effort of creating something for community and you're uploading it for goodwill, and I'm just making malicious things and calling them similar names to yours, you know, for every good thing that happens, there's always gonna be bad things. And I think the same's happening now with ai, the difference now is the development is so fast.

I don't think people fully understand what these models are, the data sets associated with these ideas of skills that if I want to play around with ai, I'm just gonna grab everything I can find off the internet. And of course I have no idea what I'm downloading at that point unless I actually understand what these models are. So what are we supposed to do about all this?

Because I don't think we can put the AI genie back in the bottle at this point. So are there things that organizations need to implement, especially say the DevSecOps teams that, you know, are ultimately responsible for all this stuff? Um, and and and how hard will it be to do that?

Yeah, so I mean, there's a lot of things we can think about. I always think about having like clear controls that are consistent and static in some way. So policy is always a good one.

So, uh, a type of rule you can set in your organization is like, again, this is where someone like Cloud Smith comes into place where we say, if you are pulling everything from your own private registry, assuming you audit everything that goes into your private registry, but at least you can put the control there and say, well, we only allow our developers to source from this private registry. So this idea of everything coming from public registries and we don't know what's in them is a real problem because we don't know what we're sourcing. At the end of the day, we don't know what's running in our production systems.

That's the world we don't want to live in. Um, then aside from just saying I wanna source some private, you also want to, as part of that registry control, you wanna have some kind of automated guardrail in place that that's going to do the scanning to tell us what is or is not defined as good. So when it comes to ai, again, whether you think of something like, uh, hooking face, we have these models and data sets and we have essentially, you know, Python scripts at the end of the day to do this statistics, um, we just have to understand, well what of those have certain file formats, which file formats can, for instance, be executed?

Should we scan them to see if those executable files are potentially containing something as deemed malicious? So there's all sorts of open source projects that have existed. You can think of the, you know, AVS of the world that could scan and say, I see a signature that I don't trust and that is therefore bad.

But then there's also projects like, um, the open SSF you know, they've put a, the, uh, malicious packages project in place. So with that, they will define what are or not good malicious things, um, from these public registries. So when we have something in our private registry, we want it to automatically check against that open API to say, okay, you have a package in your private registry, but it's deemed malicious by this organization.

Do you wish to continue with it? Of course you would say no if it's deemed malicious or at least with a high level of scrutiny. Um, and we need to think about that as well for all these new AI technologies that keep appearing.

So, you know, open, uh, source malware, which is one of the things to open SSFs project, they now deem these AI skills in the same category as software packages that come from public upstreams. They deem it in a case that any person can create a skill, therefore every hacker out there can also do the same thing. And because of that, you are now in a, a wild west of skills that you don't understand how the code works.

How could you possibly know if it's malware or not? So yeah, we definitely need kind of static guardrails in place to say, these are the things I have. Check them against public APIs and say, do we know if it's safe or not?

And then how do we proceed? Otherwise, we're just consuming from anything anywhere. How do we have this conversation with the developers, many of whom are, shall we say non-conformists and have skills and they'll just spin up a server in their basement and start writing some code on something and then when they like it, they'll upload it into something.

You know, that feels more like a standard corporate DevOps workflow. But you know, as far as anybody knows, the, you know, it was created with some level of alleged oversight, but maybe not so much. So how do we kind of get everybody on board?

Yeah, it's true. Like I, I love Anthropics wording when they talk about plot, where they say essentially it's not truly age agentic because even though it can do things, there should be a human that's in control of this. At least from that level of control.

They're the ones making the ultimate decision. Do I apply, do I even push this thing into production? It's not hands off.

Totally. Um, I think those projects that we saw recently, the likes of Claude Bot, this fully agent system, um, I think those are still very much pet projects for now. To your point, anyone can have a home lab running in their basement.

I can run this tool as open source, completely free. I can run on a a Mac mini. That's what everyone's gonna see buying to run it on its own and lightweight.

Um, but when we do that, again, it does or doesn't do what I expect it to do, would I replicate that thing in my home lab and then put it in production for my company? Probably not. You know, and I think that is where we're at today.

It's that a lot of these things that seem scary, are we actually using them production? I think we're so far away from that today because we have to look at non-deterministic systems and think, okay, how is it going to be used? A, a perfect example is you want a consistent response to happen to customers every single time.

That sounds good. Well, just by definition of non-deterministic, it can't determine that it will give that same output unless we put some human guardrails in place, like system prompts to say, in these conditions, always give the same thing. But when you start doing that, it's going back to what we were already doing, which was coding and saying, here are the hard coded rules of how you respond to things.

So I think organizations, developers still have to go through heads of engineering, platform engineering, whoever it is that sets the controls and standards and they drop at the end of the day, the design for how we're gonna use these AI tools. You know, something like a clo a Claude, it makes sense. It's something that's running in your environment to help you be more productive.

But in regards to these agentic systems, I think companies really have to look at themselves and say, what are we trying to build at the end of the day? What problems are we trying to solve? And if it doesn't solve a problem, how do we actually go about using or do we use it at all?

Um, but the security is a big part, you know, until we know how we can consistently secure it, can we really put it into a, uh, a highly regulated environment? Probably not, you know, so, so it's, it's an interesting problem. So Does that create something of a paradox where we're writing more code than ever faster, much of which after it is checked in, isn't making it into a production environment?

Because once it gets reviewed, it's determined to be, have all kinds of flaws in it, and so we just kind of create this endless hamster loop to no point. Yeah, I I absolutely, you're hearing more and more companies coming back with that feedback of, okay, we fully speed up process. Like I don't think code writing code was ever the problem.

I think it was designing architecture that matters and is efficient for organizations. So yes, you know, these a toolings create more code than ever before and faster than ever before. But at the end of the day, if something goes wrong and you haven't wrote the code and you don't understand how the code works, you're gonna spend twice if not longer, as much time, um, reviewing it, making sure, okay, how does this nested loop system work and why does it work the way it does?

Whereas when you wrote it and you didn't use an AI assisted code and then someone said, well, this function's not working, you can say, well, I, uh, I wrote that function and I know why it does this because I wrote it. So that is a big problem. I think more and more organizations will only learn from experience where you test it and you say, this workflow actually wasn't as efficient as we wanted.

And it also comes back to the same problems we talked about there with the ai, which is, you know, what are we trying to solve at the end of the day? Was the issue writing code or was the issue writing secure code? Um, I can still see how AI is gonna solve a lot of problems, which was in the past there was this tech debt problem that we write code and we can't write all the code out there.

So we use software dependencies and libraries from public sourced registries because there's no way we could write every single thing we want into our code base. But if you could use AI to write, essentially rewrite, you know, some of that functionality without needing to use libraries, I can see a world where AI is gonna reduce the amount of vulnerabilities in our code base. If we have implicit design structure in place where we start thinking about, okay, how can we make our code more secure using AI to actually generate that code part, you know, 'cause again, writing code was never the hard part.

It's just riding at a pace, doing it efficiently is, was always a trouble. Will we create AI agents to essentially manage the AI agents that are writing the code? And that ultimately may be how we lick this problem because we're gonna have two sets of AI agents, or maybe three or four doing different jobs, but we're gonna have to surround the AI agents writing code with AI agents to help manage and ensure that their workflows are followed properly.

Yeah, again, that, that is a world that's possible. I think it would be highly expensive as well. 'cause you gotta think about as well, if we're using these tools that have an API and we have rate limiting, or we have, for instance, we're charged for the credits that we're expiring, um, can we build a whole ecosystem where it's just agents that we're paying for to do that?

Uh, probably not. Um, I can see a world where we have more considerable design considerations. So we say, look, um, we are building a code base where we only want to use these programs, these dependencies, the things we trust.

Again, that's something that Cloud Smith can solve, where we say, well, developers only source these things and anything else, we don't trust it 'cause it's not part of our original design. Um, and then there's other static controls in place where you say, for instance, whether it be a policy in Cloud summit or a policy elsewhere saying, we only trust these file formats, these dependencies, these whatever it be, you are made very explicit about the things you do and don't approve everything else. Again, implicitly would get denied.

Um, I think that's probably a better world than having the AI be the, the gatekeeper of more AI necessarily. Mm-hmm. Um, so to your point about all that, do you think that there's gonna be auditors soon who are kind of like, you know, rubbing their hands together with some level of excitement going, Hey, I can't wait to scan all this code and find out all these ways to loving new finds?

Yeah, I, I definitely think so. I, I think a lot of testbed projects are now showing up in production that, let's say there was a race, a global race to get AI tooling out there to market. So people can say, look, I am an agentic leader, please fund my project.

But you have to wonder in that pace, were they thinking about security? Was that the primary focus? Probably not.

And we've seen that with other industries as well. We see that with, uh, smart devices that with such a rush to get gadgets out to market that could monitor your health and wellbeing. Turned out a lot of those were insecure as well.

So I, I don't think it's a uniquely an AI problem. It's more of a, an industry and market problem. All right, folks, you heard it here.

AI coding tools might just wind up being one of those things. That's too much of a good thing. Hey Nigel, thanks for being on the show.

Uh, thank you very much for inviting me. All right. And back to you guys in the studio.