Shift-left or Shield-right? The Evolution of DevSecOps – Ctrl+Alt+Deploy Ep 5

Hi everyone, it's Alan Shimel. Welcome to another episode of Control Alt Deploy. This is a, uh, control Alt Deploy is a podcast we try to do every two weeks or so here at Techstrong.

And we talk about, well, it's, it's really DevOps, but it's DevSecOps, which is kind of, you can't have DevOps these days without DevSecOps. It's about security. It's about how we're, how we're writing and deploying and running software these days.

It's, it's one of my favorite shows of all the things we do on Techstrong. It is, uh, sponsored by our friends at OpenText. So many thanks to them.

But, um, it's, it's our show. It, it's a tech strong event, a production, as we say, and, uh, have a lot of our tech strong friends on this particular episode. I'm looking forward to it.

Today's episode is titled Shift Left or Shield right, the Evolution of DevSecOps. And, and that's a loaded question we're gonna have a lot of fun with. Let me introduce you to our panel members for today.

If you watch Techstrong Gang, you've probably seen a lot of these folks on, on the gang, so they may not be strangers. Gee, I'm gonna start with our, our friend Kate Scar, and welcome Kate, if you could give people a little bit about you. Sure.

I've been, uh, part of, um, I've been doing technology since 1998, started with IBM and again, you know, cybersecurity with us. Started with network security, AV and dare I say Tivoli identity and access Management. So, Ooh.

Yeah. Oh, that. Hey, t's gonna rule the world.

Thanks, Kate. Um, joining next is our good friend, Tracy Reagan from Deploy Hub. Hey, Ellen.

Hey, you know, Tivoli used to have some pretty righteous parties in Austin. All I have to say about that, and yeah, so I am Tracy with Deploy Hub. Um, I do get to enjoy being on the gang, uh, on Mondays, which is a lot of fun.

I'm part of the Linux Foundation's open source security foundation, um, board governing board, as well as a continuous delivery foundation's board. And I'm really into open source and I'm really into fixing post-deployment vulnerabilities. Excellent.

Welcome, Chay. It's great as always to have you on. Next up, we have an analyst, gang member tech Field day, uh, delegate.

Our good friend Jack, Jack Poller. Hey, Jack. Hey, Alan.

Great to be here. Uh, I am the founder and principal analyst for Paradigm Technica. I have a long history in technology, a few more gray hairs than Kate, and a few more years.

Uh, I started as an engineer, turned into a marketing person, and then an industry analyst focusing on cybersecurity. Excellent. Thank you, Jack, and welcome.

It's always, it's always great to have you on. Next up, I wanna introduce you to Garima ba Baal. Uh, well, I'll let Garima introduce herself.

Garima, go ahead. I'm, I am based of AWA Canada. I'm the founder for the DevOps Committee of Practice here in Canada.

I just, several chapters. I'm also the chair for the ambassador program at Condenses Delivery Foundation, written several books, and, uh, my latest book, which is coming out, is Mastering Security at Scale. So hopefully I can value add to this panel.

Oh, I'm sure you will. Garima you always bring value to every, every panel, every show we do. So thank you for all you do.

Last but not least, he's, he's the newcomer to our group here today, but we're gonna not hold that against him. Trey Island. Trey, welcome.

Introduce yourself. Uh, Thank you very much. Yeah.

Um, I'm based out of Denver, Colorado. I'm a security consultant. Um, so that means I am the technical hands-on demo guy, uh, when it comes to, Hey, how do you integrate application security into your organization?

Are you ready to move to the cloud? Or do you have CICD implementation? So I kind of help with all of that.

Uh, integration with our tools for scanning the source code mobile applications, uh, open source and dynamic scanning. So guys, let's dive into it. com because of what became DevSecOps.

I thought DevOps was gonna give us a chance to do security better, to correct a lot of mistakes that I had seen, you know, in my years in security. Uh, we didn't call it DevSecOps, truthfully, it was rugged DevOps. I remember the fights I had with people in security and the people in DevOps because there is no, there's just one DevOps, you don't need a second there.

You don't need biz in there, you don't need anything. The security people said, uh, you know, it, it should be SEC DevOps because isn't security first always. Um, and then we, you know, this whole idea of shift left, and I was, I was so gung ho from s**t shift left.

I believed in shift left from the bottom of my heart. And it, and it, you know what, over the years caught on DevSecOps became a real thing. Most of the DevOps companies considered themselves DevSecOps companies.

We shifted left and we shifted, left some more, and we even went a little further left, and some began to question, did we go too far left? Is it really working? Maybe we should shift right?

Shift up, shift down, shift everywhere. We still need better security. Kate, if you don't mind, I'm gonna ask you to kick us off here.

Did we shift too far? Left What shift left the right move? You know, one of the problems that I, I, I feel like we continue to have is that it, I think, think originally it was a good idea to shift left because the people who were coming out of, um, school, they, we just weren't, it wasn't being taught.

So we had to start somewhere in this and shifting left and trying to add security because we were being hit. I mean, I still remember, you know, the SQL injection attacks in, you know, 2003, 2004. I mean, it, it was, it, it was taken us by surprise, right?

And I think at the end of the day though, we still, you know, we became cybersecurity people became these roadblocks and to business and to the dev people. And, and we were really putting a lot on application teams when they weren't security people at the end of the day. So I, I think we did go too far, um, to the left.

And I, and I think that we didn't work together. We put a burden on them, but we didn't lift a burden and we didn't share that burden going forward. So I think it's better that we are starting to look and, and create this culture of let's really take a look at this because we all want, um, we all wanna do it safely.

I mean, at the end of the day, you know, it, it's, we have to be better at working as a team. Yes. The team thing, Reemer, Go ahead.

The team thing, they're both, yeah. Yeah. That, that team thing is so important because, you know, it's, you know, I, I was doing software configuration management and the late nineties, all through the early two thousands, and I never even talked about security.

I never even heard about it. I just thought security was something was done behind the other, the, the curtain oz was back there dealing with security, and we didn't have a discussion about it. There were, there really wasn't any, any tooling to add to anything that we were doing that would improve security.

So shifting left was, uh, a, a shock when suddenly we were told, oh, the development team and your, um, your, your SCM at the time needs to have more security in it. We were like, well, what kind, what do we need to do? And in fact, that was the first time we started looking at, uh, what they call software de bloating now, um, to shrink what libraries we were pulling in it in a shared library environment to try to minimize the amount of libraries that we were bringing in so that we could do better security on the, on the binaries that we had.

So it didn't have so many executable, uh, functions in it. So, you know, it's interesting that you say the team part. 'cause I think that's where we got caught up in the beginning and suddenly it was securities got oz, but then you're gonna have to shift it over to the, to the, the munchkins to get the work done.

And we didn't know what to do. Yeah, yeah. It really wasn't being taught.

No, not at all. It was security was not taught to developers. That's for sure.

When you have computer science reemer, you know, the voice of DevOps here, shift left was such an important piece of it for me. What about you? It is still an important piece, but what I feel in today's AI era, it is shifting, uh, from a personality perspective, which is basically having more, uh, and new components of, you know, how to integrate security when you are looking at the development stack, because a lot of developers are using AI and AI native tools to kind of in, you know, build code and, you know, also develop and review and test and deploy code, right?

So there are new types of security, uh, you know, is required and new, new type of security vulnerabilities are introduced in the code itself. So shift left is changing, and, uh, obviously, uh, there is a lot of upskilling required in that dimension. And why runtime security is important.

I'll put some facts on the table so that, uh, you know, we understand the urgency of it. Uh, there was a report from Checkpoint, which says that, uh, every prompt, which we do, uh, one out of 80 prompts are posing higher risk of, uh, sensitive data leakage, a hundred compromised AI models were deployed into hugging face platform, which is basically for a lot of people who are using it. And there is dark LLM, you know, the malicious modification of AI models, for example, is happening as we speak.

So if you think about this shifting left had reduced the vulnerability problem by 70%, right? 30% was still runtime security gaps, which we were finding. But now with the injection reduction of AI into various, uh, SDLC lifecycle phases, it becomes more urgent to ensure that we don't look at only runtime security, but also looking at shifting left and seeing what kind of new vulnerabilities are getting started through a AI injection.

I can talk a little bit more about it, but I think from a community point of view, we are seeing a lot of these things which are, which needs upskilling and, uh, I mean, this is a bad news that, you know, uh, we don't have enough talent, we don't have, uh, enough education and awareness and this dimension and where there, where the communities like this, uh, w we drive come handy and we foster that collaboration. Excellent. Gima, excellent.

Jack, Trey thoughts? Well, I, I may, I don't know if I'll be call it controversial, but I have a slightly different opinion, which is really embrace the power of, and rather than, or which is, I think we need both shift left and shift, right? Which, you know, defense in depth, right?

We are having different types of controls at different points in the process and in the life cycle of the application to solve different problems. Shift left is really, you know, Kate talked about it not teaching cybersecurity to, you know, early engineers, but even senior engineers who know about cybersecurity don't address cybersecurity because functionality, feature functionality and schedule is the most important things to the company, not security. And so we, that's how we measure our developers and our development life cycle, right?

So Shift Left is a way to introduce cybersecurity into that conversation, to bring it level of importance up so it gets addressed as quickly as possible. That doesn't necessarily make it sufficient to protect our applications. We also need security shifted, right?

To do more at runtime, to catch things that can't be caught at the early stages of development lifecycle. Fair. Re you're talking to real life customers, users.

What, what's your view on this Shift? Left is very important. I think the problem, the problem resides when security then offloads their responsibilities onto the developers.

And the developers then decide what security tools they want to use because of ease of use. Not necessarily this tool is better than the other. I've seen a lot of that issues where developers then have a lot of power to dictate what security tools will be used, but they don't really have metrics of why other than, oh, this, look, this works really good in my IDE as far as usability, but what security checks are in place.

I see a lot of that. Um, now with these AI tools, it's gonna be up to security to continue to research and understand these vulnerabilities. I think it, for me, my background was, I was a developer before I became a security analyst, before I became a security consultant.

So I'm kind of able to have a conversation at a lower level rather than just, Hey, go fix this because the report says, so that I think is a lot where there is contention between developers and security analysts. 'cause the first thing a developer will say, okay, can you tell me why? Or do you, my, my application works like this.

Why is this a vulnerability? You can't just say, it's only in the report, go fix it. You have to go on that other level.

Um, so that's where security is gonna have to continue to do their work, their research, their efforts. And I see AI as a complimentary tool. Um, the problem I see on the development side, if it continues to go down this path, is this whole thing with open source, right?

You have something in your code that you did not create. You don't have a good understanding of it. And if you're just gonna to use these AI tools to generate an application, you're not gonna have a good understanding and you're probably have a lot of loaded code for functionality you didn't even need to utilize.

So that's where organizations are gonna have to lock down what tools that they allow Code let's you have insecure code. But go ahead, chase, go. I'm sorry.

Let's Talk about, let's talk about tools for a minute. So I just spent the last week we have the, at the CD foundation. Kate and I are on a, a special interest group called the CICD cybersecurity ums.

And we have a deliverable, so I, I gave up this last week to start working on looking at tools and how they fit within the secure software development framework. And I'm not gonna say AI's out there, and it's gonna probably change the way we do things, but there are so many tools today. I am, I was shocked by the number of open source tools that have been delivered to the industry that I know we're not, we're not using yet.

Not everybody's using them or taking them serious. It it just look at the problem with generating SBOs. Not everybody generates an SBO M1 of the core components of your secure pipeline.

So we have to remember that while we have this shift left discussion, and many of these tools are on the left side of the house, there's also many that the platform engineering teams are gonna start using that are sort of squished to the middle. Yeah. And the, and many of those ones in the middle are, are actually starting to monitor what's happening in production.

So maybe we've come to a place where we're shifting. Um, we're, we're shifting a a lot of tooling into the middle that catches things as it's coming through the pipeline, if they're adding it and it's starting to monitor what's happening in production. I really was surprised by the number of open source tools and the, and the security features that these tools offer that can fit today without any ai, without any new, new tooling to solve some of these problems.

Um, and I, you know, I hope when this document gets out that people can use it as a research tool because it is shocking. I mean, I, I was thinking I'd have five or six tools per category, and I'm looking at 25, 30 tools per category. Wow.

All of them doing something a little different and solving the problem in a different way that they do relate specifically to the challenges that have been brought up in this, in the secure software development framework. And it's a really good guideline to use that framework because it gives you a real, a clear indication of what your goals are, but it doesn't tell you how to solve them. So what we were trying to do is say, here are the tools that will solve these.

And I were shocked. I was really shocked. It's taken me all week to get just a few of these pages done because there's so many tools and sorting out what they do to fit that has been a challenge.

So I'm hoping this helps. I really do, because we don't need to wait for AI to solve the problem. There are tools out there that can do it today.

Yeah. Yeah. And, and I love it.

I think you used a key word, um, platform engineering this idea about that, right? It does come to that middle. It, it, it really, um, I think it's a perfect word to that encompasses, um, everything that we're talking about from the shifting left to the, you know, runtime application protection.

It, it, it gives this whole more of a holistic view and I think where organizations are, are moving and it's better. Um, I do wanna address quickly, if you don't mind, uh, with Jack this defense in depth. You know, it's something from a strategy point of view that I have seen that really isn't working.

And the reason being is that it almost creates more of this whack-a-mole type of strategy where you get a, a vulnerability and get a tool and you hit it. I think in what we are trying to work on, um, with, with Tracy, um, is more of this holistic type of picture and a strategy that is more proactive instead of like a proactive offense, more so than a strategic, um, defense, which is different when you think about it. You know, you still need to have an offense strategy.

It doesn't mean that we are going to attack. It just means that we're setting ourselves up in a position that we understand, hey, a heavy hitter is coming to, um, to hit, are we gonna be all in the infield or are we gonna go to the, you know, off field and get ready because we understand that it's coming. We know the threats, we understand the attacks.

There really isn't anything new even with that ai, they're still the same attacks. We know this. And, and so, um, with the tools that are out there, some phenomenal tools like Tracy is saying, it's, it's, it's, it's such a beautiful time to be a part of cybersecurity.

I, I, I'll, I'll tell you, I don't disagree with you at all. I highlight defense in depth more to highlight that a single tool is not a silver bullet, right? That we are not that simply doing shift left and doing static code analysis or dynamic code analysis, whatever your shift left or combination of shift left tools is gonna give you isn't going to solve or, or provide you perfect security.

Right? And I think you mentioned in the word holistic, which is right, is that we want to think about the entire gamut of everything from the very start of the project architecting security into the design, through the coding phase, through the test phase, through the deployment phase, through runtime, and then even how do you end of life the product and how do you secure it, right? Yeah.

And what do you do with the data at the end? It's an entire picture and there's an entire set of problems. And one tool or one small set of tools shifting left is not going to solve our problem.

So I'd like to people to think about it as, and, and I, I appreciate the, the, the, the analogy of whack-a-mole we do in cybersecurity, spend a huge amount of time doing whack-a-mole, which is, I believe the wrong way to do it. And I think the right way to do it say that we have seen these problems in a slightly different domain. AI is a brand new domain, but it is still a data leak problem, right?

And how do we treat data leak problems and can we, uh, uh, repurpose tools or apply the same tools as Tracy said, where you said there's hundreds and hundreds of tools. How do we use these tools to solve that problem without saying, oh, we have to wait for ai. Yeah.

I I would also like to shift this discussion to runtime security and, you know, uh, of, of course there's a majority of work which is needed to be done in terms of, you know, securing the legacy or securing the as is or status quo situation. For a lot of organizations, you know, there's a maturity curve. So a lot of organizations are already behind, right?

So the 70% of vulnerabilities, which can be found through injecting security through shift left is not already happening. So that addresses or caters to that. But if you think about runtime security and why it is becoming more and more important, and the CXOs have a shorter runway of 36 months to prove this because AI is coming, and I'll highlight three points.

LLMs, you know, you, like it or not, developers have started to use LLMs in many shapes and forms. So the LLMs are creating code, right? The second part is prompts.

So we all use prompts, right? And if you think about what tasks software engineers are accomplishing through prompts, there are many, right? So test case generation, for example, uh, has a high kind of volume where, you know, people are generating, uh, test cases through prompt engineering, right?

So, uh, the third aspect is AI agents, you know, if you like it or not, the AI agents are coming in the operation stack as well, and they, they're using LLMs. So for these three special components, which AI is bringing, we need a special, uh, security mindset. We need to have, you know, specialized components and security guardrails to not to inject malicious code, for example, uh, data poisoning through prompt injections.

Even AI agents, they are playing a, uh, a bigger role because a lot of autonomy and decision making is happening through AI agents. So it is more and more important that, uh, people start to invest in runtime security. Uh, I don't disagree at all.

I, you know what, I, I like the term shift everywhere. I, and I, it's not my term actually. I first heard it from my friend Jeff Williams from Contrast Security, right?

But certainly we've gotta shift left, but we can't expect our developers to become security Pros, right? As Trace said, they're going to, they're going to lowest common denominate a least path of least resistance, whatever one's easier for them, whether it's good security or not, it's something, but we do need to have runtime controls. We need to remember that security doesn't end at the deploy button, or we don't actually press a button for Deploy anymore, do we?

But it doesn't end at the deploy that that mission continues as well. And, and so it, I would like to see a holistic security view of, you know, throughout that the life cycle, not just of software development, but of software operations, right? Observability and security is, is something we haven't talked on here, but that needs to be part of this as well.

Um, I, you know, we, security's important and no matter who you talk to, I think no one says, ah, security's not really important. We all say it's important, but we can't just focus on the security over here or the security over there, or at this stage or that stage. Every stage needs security.

And I, I think the, one of the problems with security left is we took our eye off the ball of right. And runtime and, and these other, these other places, Uh, know being a, you know, I wanna, I wanna, I wanna disagree with that statement just for a minute. Go ahead.

Because, you know, if you look at what the open SSF has done, which I work with quite often, and they talk about security all the time, there has been a quite a bit of work done on trying to create that holistic view. That's why I'm gonna push again, if you have not read the SSDF, this is a, this is like a reminder to do that because the goal was to create that holistic view and there has been a ton of work on creating that holistic view. So read the SSDF because it's, that's what that is.

I I will and I should. And, and Tracy and Kate, when you guys do finish this deliverable here from the, uh, CDF, I'd love to have it either on one of our tech strong properties. Let's get you both on and, and, you know, shine a light on it because it sounds interesting.

Trey, actually we have in October. October, alright, I'm marking it down. Trey, I feel like we haven't heard enough from you on this.

What are you, what are you making of this discussion? No, absolutely. With runtime, right?

You have no, you have an idea of how your application should run when it's under a load, when users are actually actively using your application. But there's always that use case and sometimes it only takes one to break your application or have data leak. That's why it is important.

It's not important. It's important to have these tools, right? But it's also, why do we have these tools, observability, what are we doing with that data?

Who's managing that data? If a tool is fa failing, what is the corrective action, right? It's all these things you just can't throw.

And like, uh, Tracy was saying, there's so many tools out there. How do we actually, um, identify the ones that are correct for our use case? There could be a tool that's gonna be great for one company, does not mean it's gonna be great for our company or our application.

So that's where a lot of that research does have to come into play. Um, and having information on the log injection, how is the host running? All of that is important of course, after the development phase.

But if we can do that in every phase development static, well, static analysis, dynamic analysis, how is running that is gonna give the holistic view, but sometimes I see is there's so much on dev teams to do almost all of that. And they're great at developing code now you're forcing them to put another hat on, another hat on. And in my role in the past, because I'm a jack of all trades, I enjoy learning things, but I'm not ne I necessarily did not have teammates that had that same, uh, go get it mindset.

And then you feel like you're, ah, my last name. Like you're on an island all by yourself. Um, Yeah.

And, and there is that, that we need I'm sorry, go chase. I have One, one, I it based on what Trey just said, something came to mind what companies can do to start understanding their gaps in their shift everywhere approach is they, like we, we did in chaos engineering, we need to start doing game days where a, a fictional, uh, you know, software supply chain, CVE, that's critical or high risk is floating out there in your live environments. Watch to see how long it takes your team to re respond to it.

What is your meantime to remediation? Those are the kinds of things that organizations should start looking at. Uh, because I'm, right now it's over a hundred days.

We've gotta get it down to less than 15, less than 10 would be good because it only takes 10 to exploit. But we ha we are over a hundred days folks, and that doesn't work. So game days would be a really important, um, exercise for your team to start practicing because it means every single person in the organization from developers who have to recreate the, the new palm files all the way out to the deployments have to, that that whole, that whole cycle has, has to be hit when there's one vulnerability that has to be fixed.

Great. Hey, Jack, I'm sorry. Go ahead, Kate.

Oh, I, I was just gonna say I'll, you, I'll come back. Jack, go. So, um, so quickly, the only thing that I'll, I'll add is that, you know, it's not as bad as it was meaning, um, you know, when we used to go talk to application teams, teams there used to be like, you know, what are we talking about?

Like, you have no, I like, and there was such a pushback. You don't see that today. Today.

You actually have people who are interested and, um, who are concerned and still feeling overwhelmed by, by all the different tools that are out there. And I, and I think, um, and, and I believe the way that Tracy, you know, broke things down very easily, um, within this deliverable, I, I believe that it will help. But making it simple, I think will, will go a long way into making sec important in DevSecOps.

Go ahead, Jack. I'm sorry, I I don't disagree. Jack, when you talk to consultative clients, right?

Analyst service, do they take this? Do they ask, do they want a holistic approach that shift everywhere or do they focus in on a particular stop along the SDLC? I think they, right now, vendors are primarily focused on a particular stop along the SDLC because they perceive that as a way to market and sell.

Not that that's what's really needed. And something that Kate sort of said resonated with me. Part of what I see and what I bring back to vendors is when I talk to practitioners, they complain that the security tools are built for security people, not for developers, right?

When, when I was a, early on in my engineering career, I started out as a software engineer and then I went and started developing chips. And one of my mentors in the chip development space said, well, all the code you wrote for the chip will work, but you write it like a software guy, not like a hardware guy would. And it took me a long time to figure out what that meant.

And it's really you, the way people do things and operate in DevOps is a different mindset comes out. You start with different assumptions, different perceptions than you do with when you start out as a security person. And think about it as a security person, I think the security tool developers need to put themselves in the position of the practitioners and have people like Trey with them who can represent the practitioner point of view and say, this is how we really use that type of tool in our environment.

Build it for us, not build it for you. And I think that will really help Build it for us, not for you. I think that's a great place where we call pull the plug on this, Jack.

It's a good, good way to end it. Build it for them, not for you. Kate, Tracy Reem, or Jack Trey, thank you all so much for joining us.

We, we try to keep these to a half hour. We're a little over, but we're closed. Many thanks to OpenText for their sponsorship of this and all they contribute, so we appreciate it.

Many thanks for you to, you guys for watching. We'll be back in another two weeks with another Control alt deploy and we might be doing some more live round tables where you can take part in them as well. So stay tuned for that.

Until then, for Control, alt Deploy and Techstrong, Ms. Allen Shemel, we're out.