The Pitfalls of AI Code Generation – Amy Baker, Security Journey

This is Textron tv. Hey guys, thanks for the throw. We're here with Amy Baker, who's security education evangelist for Security Journey, and we're talking about the impact that AI is gonna have on DevOps and the quality of the code movement through those pipelines.

Amy, welcome the show. Thanks so much Mike. I'm really happy to be here.

All right. Just about every developer these days is at least relying on some form of ai, perhaps the right code faster. I'm not sure it's any better though, but, and I guess that brings us to the question is what is the ultimate impact on all this stuff?

'cause we are certainly seeing developers become more productive in terms of lines of code, but what does that mean? Yeah, it's a great question. You know, of course as technology advances we all wanna take advantage of it.

And so I think it's terrific everyone's diving in and looking at this, but there are studies out there that are really showing that what's happening is, is the developers are writing code more quickly and more confidently, but with more errors and more security flaws. And it's because when you think about it, the AI is trained on incredibly insecure code, number of vulnerabilities has increased by 60% over the last year or so. And with that as the, we have more and more vulnerable code, we are, we've trained the AI on vulnerable code and that's what it's gonna produce for us if we let it.

Does that mean we just need better AI models or the ones that we have are kind of brute force, they're general purpose and they hoovered up everything in the world and we got to the average, which isn't so great. Well, right. It's a garbage in, garbage out model when it's a large lang language model and it's learning off of things that are insecure inherently.

And I, I would say, you know, in general, insecure code is still a massive issue that's not been solved. And honestly it's really, it's an interesting dilemma that we've put the developer in because unfortunately they come out of university, they come out of their training programs without very much knowledge on secure coding. They can write great functional code, but they've not necessarily been taught how to write secure code and it's something they have to learn on the job or their employers have to support them with in some way.

And I think it's not well understood that it's a written employer's responsibility to educate what could be a pretty highly paid employee on that part of their job. And I think that's part of the challenge is because there's so much insecure code, what people learned from is learn from a base of insecure code and we're not necessarily making progress on that change. Do they even understand how the code works if we're just kinda ultimately starting to copy and paste things and moving things around?

We've been kind of trying to make developers more security conscious for a while now, but it seems like they're moving away from understanding how the interest of the code work and that's where the vulnerabilities are. Great point. Absolutely.

I think there's a piece of this where they are, um, I think there's a great application, maybe we should start there. I think there's a great application for very basic, um, functions. Um, great for maybe junior developers, but a more complex problem or more experienced developers are going to be doing more complex things, in which case I don't think the AI is a great solution for them at this point.

It still requires some understanding. Um, there's a Stanford study actually out there that's really focused on this and, and actually is part of what's proven out the, they write code more quickly and are more confident in it despite the fact that it's gotten more flaws. And I do think there's an area there where the developer knowing that they really wanna be great at their work, can't use the AI as a crutch, they're going to have to grow their knowledge of this topic and really start to excel in these areas.

And I think so often today people have an exposure to the OLA top 10 and maybe the C W E top 25 where you understand what the vulnerabilities are, um, that are out there, but it's just an awareness and it's not an education. Um, so just like so many other roles that require deep technical knowledge, there has to be an educational program that supports that. It can't just be an awareness video and call it done.

What will be the impact on DevOps teams and the security folks? 'cause it seems like at least in the short term, there'll be a lot more insecure code running through the system that somebody's supposed to check through a DevSecOps because uh, I can tell developers all I want, but changing developer behavior seems a little difficult. It can be a little challenging.

I agree with you. The good news about them is they're super confident in what they do. Right.

Um, so what I would say is there are a couple ways that people are are, um, obviously today trying to validate that they have secure code, they're using code scanning tools, they're using code reviews. And both of those really obvious practices, they um, they find the vulnerability but they don't fix the vulnerability. And very often when it comes to code scanning, it's actually a false positive, in which case now you've provi presented a challenge to the developer, they look at it and say, that's not really a challenge, and they move on.

And so I think it makes them somewhat more numb to the problem, which is not great. The code review piece also relies on knowledge. So if you're doing a code review that's either a manual code review or something that's a peer code review, once again, I'm back to, wow, somebody has to actually know what insecure code looks like in order to identify it and do something about it.

And so I do think as we're building more and more insecure code more will slip through if we haven't solved that first problem of you have to have knowledgeable secure coders to begin with. Is this a short term problem? Will we see LLMs that are trained on specific domains and things will get better eventually?

Or is this gonna be a more permanent state of affairs? Great question. I mean, let's be honest, we've not really made great progress on a WASP top 10.

So they did a rewrite of the, you know, common application security issues. Um, they moved from 2017 to 2021. All they did was broaden the categories.

They didn't remove any. So in more than a dozen years, we have not reduced the vulnerabilities. Those things that have been around for more than a dozen years are still around now.

So what code base will they build the AI from that will be secure? Gonna have to start at the very beginning, I suspect. Um, is it your sense that the bad guys are kind of looking all of this and kind of laughing and saying, this is great, they're building more insecure applications than ever, and away we go?

Well, I mean it's almost worse. I think that's true, but I think it's almost worse than that is they're looking at all this great AI software and saying, well, great, that's not secure either. So now I can feed all of my sq, I can feed all of my exploits and all my vulnerabilities.

I can feed a whole bunch of great things into that AI model and make it worse. And now there's a big call for cisa, just put out research on securing AI and securing these models. Because if somebody can hack it and deliver results to you that are producing even more insecure code that they can exploit, the problem just continues to get worse.

In my mind, I have to wonder, so we're gonna create LLMs to write code that are insecure. So does that mean we need more LLMs to keep track of the insecure code being generated? So now I got LLMs monitoring LLMs for insecure code.

Does that make sense? Yes, it does make sense. And it's also, of course, I think what most people are so concerned about when it comes to AI is when does the machine take over and where are the people, right?

Where are the people who can stop? And I, the analogy that I came up with as I was thinking about, um, our discussion today was, have you ever followed your g p s blindly and has it ever led you to a dead end or to really bad traffic or to a whole bunch of things that you probably should have said, well, I really should have looked at that before I followed this blindly. I feel like writing code that's based upon AI output is the very same thing.

Don't follow anything blindly. You have to, we have a human intellect for a reason, right? We, we all have to inspect and review and understand exactly what these things are doing before we just follow.

I feel like there is also not a universal respect for high quality code. A lot of folks are just in the mindset of, you know, crank this thing out and if it works, great 'cause you know, we have a million CPUs and no one needs to care. But it does seem like, you know, there's a lot of technical debt in this conversation of which security is only a small piece.

So do we not have the right appreciation for what's going on with those lines of code? It's a great point. It's interesting.

I had a conversation with a CISO not long ago who basically was introducing concept to me of elegant code. Um, and that there was a lack of appreciation of developers who can write elegant code that's concise, well thought out, easy, easily documented, easy for somebody else to pick up and follow, and that maybe this particular practice doesn't get the respect it deserves for and, and is not being encouraged for the quality it really needs. I mean, you think about it, the white, um, the White House national, um, security update that they recently made.

The focus was really on pushing the liability for software, um, insecure software to the software developers themselves and the manufacturers. So trying to take that responsibility off the users of the software, which they can do very little about it and push it back to the people developing software to say, look, you just have to get better at this. Like, we can't put this at the feet of the user.

It's not their fault. You have to produce more secure software so that everyone can use it safely and know that their data's being protected and, and they're not, you know, at risk of an exploit. What's your best advice to the cybersecurity team?

'cause the truth be told, there's often not a lot of love loss between security folks and developers. Developers are trying to end run the rules as much as they can. The security people often view the developers as the root cause of all the original evil that they have to deal.

It's a great point. I mean, I do think we have to give, um, the developers a little bit of grace. They've come into these roles without the right skills and security, let's be honest, has a huge shortage themselves of skills.

And so I think organizations, the way to bring those two teams together is to appreciate that they both need each other. So if you take a developer and you build them into a security champion, if you find somebody that's passionate, you find some percentage of developer audience that's passionate about doing this well, and let's be honest, anybody that's excellent at their role, there's only a, a subset of them that are excellent, they wanna do this well and so then encourage them to become security champions that can, this can solve so many problems. It can build that bridge to the security team because now I've got developers who care about the things security cares about.

Now I'm also trying, I'm also starting to prevent that workforce shortage that really exists is I'm actually building great investment in my team from within and I'm building skill and knowledge that both of those teams can appreciate together. And to be honest, there aren't enough application security engineers, cybersecurity experts, there aren't enough developers. So if you can take the people that are there and bring them together and build that expertise internally, now you've got maybe a more cohesive team so they aren't at odds with each other and they appreciate each other and the both challenges they're trying to face.

How do you have that conversation in a way the developer appreciates? 'cause a lot of the times they get alerts and then they go chase these things down and discover, well actually we're not using that particular piece of code in this module that you said was a vulnerability. And after a while they get a little, uh, tired of chasing their tail and then they stop listening.

That's a great point. One of the, um, we produce a, a piece of content to help bridge the gap between developers and security that's available on our website, but we talk about some guidance there of, hey, security before you just toss a vulnerability over the wall that youth has come up through a code scanning tool. Can you validate it?

Just validate it's legit. It'll build a lot more credibility with the things that you do share with the development team if you validated that vulnerability first and you've not just passed things along. And so, um, there are multiple other ways that we kind of get some recommendations for both developers and security working better together.

Uh, and bridging that gap and that divide, which to your point, doesn't help us have safer software if both of those teams are at odds with each other. To your point earlier about we will see more stringent regulations going forward. Do you think as part of the DevSecOps workflow, we need to start tracking some metrics that kind of show, hey, this piece of code wound up being deployed and here's what it cost us and here's what it cost to fix it.

And that becomes a number that people can kind of wrap their heads around. It's a, this actually really hit a, a nerve for me because I'm shocked by how few organizations actually understand the cost of remediating security flaws. It's, I believe it's massive.

When you look at the cost of a developer and the time it takes and the number of steps and the number of people involved in a patch, I think the costs are extreme, but I have not yet run into an organization that's good at evaluating and understanding what it's costing them. Of course, much of this focus is on applications that are being billed today. But if I think about this in a way that's kind of mind boggling, but the number of applications running in productions that have vulnerabilities is astronomical.

So I mean, can we ever go back in and fix all this stuff? Great question. I mean, I guess you have to start where you start.

Um, if you sort of throw your hands up in there, you definitely know, right? Um, I do think there's a piece of open source that people have to consider, right? So much software's being built with open source.

Has the open source been vetted? How often was it updated? Do you know it's from a reliable source?

Do you start with some of the software supply chain? There's been a ton of discussion about, um, software bill of materials and how do you understand what you're shipping to the consumer and the people that are using your software? And if there's a deeper inspection there, do we start to understand where the vulnerabilities are coming from and are we more careful about the software that we're selecting as we build the entire solution?

It's, I think it's a matter of are revetting what we're using today and, and how do we get a team of people who I can identify the problems? All right folks. Well unfortunately things may get worse before they get better, but once again, once we look in that mirror and figure out who the real enemy is, we can get started.

Hey Amy, thanks for being on the show. Thanks so much, mic. I really appreciate it.

Alright, back to you guys in the.