Crossing the Chasm of Security and DevOps | DevOps Connect: DevSecOps 2023

Well, hey, welcome to our session today. We, we have something a little different. I'm excited that you've joined us.

I'm especially excited with the, the panel that we've put together today. Um, I've, one of the things I wanted to say to kind of set up the topic is, uh, uh, every year I go to R S A C and I've been going there since I used to create cert security products, but I'm also a software person. I was wondering like, when are we gonna talk about how software gets built and the security of that and APIs and, and this year that really jumped out as it's, those conversations are happening.

Software supply chain, s o m, um, vendors I might've thought of as a application security company, are at our S Sac c maybe they were there before, but not as visible this year. Just for whatever reason seem to be the year that, um, security and software DevOps are, are visibly working together. Sort of the same thing that I think a lot of organizations are going through their own transformation.

So we have this great panel of, uh, JJ Manila have her, her introduce herself in a moment who is a security expert wire, one of the premier wireless experts in the world. We have Derek, who is DevOps security software, kind of the intersection of those things. And then Tracy Reagan, who is steeped one of the preeminent software, uh, developers software create product creators been working in, in how we create software as well as building software herself.

So thank thank you to three of you for joining. Um, I'm gonna describe this conversation as three people walk into a bar, a developer, a security engineer, and a DevOps engineer. And this is the conversation they might have.

So, so let's do introductions first though. Uh, JJ would you introduce yourself, please? Yes, I'll introduce myself.

But first I have to ask you, Mitch, if, if three people walk into a bar and there are four of us here, what are, what are you? I'm the bartender. Best job in the house.

We're not tipping you though. Well, I've heard about, about you all. Okay.

All right, everybody. Well, um, I'm Jennifer or JJ Manila. I'm the founder and principal advisor at Vision Security, where I work with organizations of all different types and sizes, uh, mid-market through enterprise on different security strategies.

Um, and primarily my history has been in network and infrastructure security, um, and enterprise security architecture. So that's, uh, kind of my background there. Great.

Jump in. Derek. Yeah.

Hi everyone. I am Derek Weeks. I'm c e o at Unfair Mindshare.

I do some advisory work for cybersecurity companies and some venture capital firms. Uh, I was also, uh, senior vice president at the Linux Foundation and a vice president at sonotype. I also co-founded the all day DevOps conference and community years ago.

Fantastic. And I am Tracy Reagan, c e o of Deploy hub. I also work with the, uh, Linux Foundation around the CD foundation and the open S s F.

Um, I have been worried and working on the, on solving the problem of tightening out the software development process since about 1995. Fantastic. And co-host of Techstrong Women and my dad a fantastic coast, which is A blast.

And if you haven't seen it, you've gotta check it out cause we have a great time. It is, it it is, it is the fastest growing, growing one of the most popular shows on techstrong tv. And, and it's understandably why, cuz it is.

Awesome stuff, great content. Um, so, so let's kind of go back to that. What I'd want to create here is this conversation about how are, are we getting in, in organizations?

Are security engineers and software engineers and people working on DevOps and automating the process of creating, uh, either the application or even the infrastructure parts of what we deploy applications on. How are they starting to, to talk together and work together more than just putting labels on it, like shift left and DevSecOps. Not saying there aren't just, they're, they're more than labels.

But that doesn't always mean something to a security person. If I say DevSecOps may not know any more than what DevOps is. Right.

So I'm, I'm curious, um, JJ if you don't mind, go first. I'd love to get your perspective cuz I know you get thrown into DevOps conversations increasingly part of it's just hanging out with me. Uh, so you have that unfortunate event happening, but from your perspective as as that happens and those conversations are happening, uh, uh, so h how, how does that strike you as a security expert, uh, in engine and engineer and an advisor?

And how do you start to adapt and groc what's going on with, with DevOps and software and all this stuff people are talking about? That's a good question. It's, I think it's a steep learning curve for a lot of us.

And you know, certainly from a security architecture standpoint, the, the role is to understand all of the inputs and outputs and, and puzzle pieces and how they fit together. And one of the things that I think maybe one of the comforting things in my experience is coming out of network infrastructure is, you know, we were heavily in the world of hardware and, and hardware has parameters around it. And yes, software runs on it, but when you got, you know, some fill in the blank switch router, firewall, wireless, ap, whatever it was, S S L V P N box, it, it had a fixed operating parameters.

Uh, and that itself was hard enough to make sure we had properly configured and integrated for security purposes. And so, you know, moving into the world where now in the networking space, we have software defined fill in the blank and everything is virtualized and, and and more programmable. We have entire, you know, I work with clients that have entire networks that we're not managing with s and m p and c l I, we manage them through a p i connections.

And so, you know, I think we're, for those of us that have been over there going, okay, well we like this world is comfortable. Yeah, we are, we're being pushed and pulled into this world that I, you know, Tracy and Derek and, and, and even to, to a large degree you live in cuz that's your, your jam. Um, but for a lot of us, it's a little bit of an uncomfortable place because in software development you don't have the same boxes we've had with some of the fixed infrastructures of the past.

Um, and so yeah, I think the conversations of educating the rest of us on what that looks like and feels like, and you know, especially Tracy with your work, what, what that program in building, uh, an application should look like in that life cycle, um, is gonna be hugely important for security professionals and architects and engineers to understand what we don't know and then the right way to do something instead of going on what we're, you know, in, in many people's cases, their exposure to it is what they have kind of witnessed or observed in passing from their peer teams, which is maybe a small slice of things and it might be a small slice of a bad apple and not the, not a best practice. And so I'm really interested in this conversation with you guys to understand what should we be doing? Well, let's go you next, Tracy and Derek, you can kind of be the inter intersection, the glue in the conversation.

Uh, and, and, and Tracy, I know you deal a lot with security. It isn't something that's foreign to you in terms of, uh, you know, software security and supply chain and SBOs and, um, but I'm guessing you probably haven't lived in the, the same world that that Jennifer JJ is coming from. What's your perspective is you talk to security people and think about the broader security, you know, we said let's shift left and make it the developer's job.

Oh yeah. Let's let them have do one more thing too that they aren't necessarily wanting to do or maybe not an expert in share with us your perspective on how all this strikes you about what we're asking from developers and what we should be doing. Well, I think from a developer perspective in the past we have relied on JJ to fix our problems.

Um, it's true. It's like, you know, security is an architecture issue. If they're getting in, they, you know, they're penetration is not our problem.

Somebody didn't do their pin testing, it's over there. But, uh, in recent, in the last year in i I call it a security awakening in terms of software development. Um, we as software developers have to be accountable for what we do, uh, and start understanding that we are the first line of defense.

It's not just when we push it over the wall and we worry let architecture and operations and production worry about making sure we have a secure environment or that, you know, our login routines are perfect. Software developers are now having to be accountable. And it's difficult, uh, because for the most part we like to, to code.

We're coders. Uh, we are, we, we like to specialize in the build, you know, creating the binaries and then automating the DevOps pipeline. And we have never addressed security in our DevOps pipelines for the most part.

And we have a lot of challenges, a huge amount of challenges, especially around DevOps, uh, to, to evolve our DevOps into DevSecOps and be accountable in the way that JJ and her side of the house has been accountable for all of this time. We're just now starting the conversation. They've been having it for years now on their side.

They've gotta start worrying about, you know, Kubernetes and API connections as she pointed out, like these thousands of microservices connecting via APIs and what are those inputs and outputs that could have some vulnerability. Uh, but it's the software developers too to be the first line of defense and that's what we have to learn. Amazing.

Fantastic. Go jump in there, Derek. Yeah, so, uh, I think the interesting thing for me is that the expectations around, uh, DevSecOps, if anyone wants to call it that or or DevOps practices is just, you know, years ago the conversations were around we have to do more to secure our software.

We could be doing more. There are things that we could work on. And there were individuals starting up efforts to, to say, here's what we could be doing.

And I think those efforts from the last five years, 10 years e even more, are now shifting the, the, uh, action orientedness of the, the developer community to say, we can do this. The technologies that we were talking about five, 10 years ago have been developed, have been improved significantly. Where they are, there are more technologies available to developers to help them integrate security more with, uh, development.

And you see it coming out from, you know, vendors like, like Microsoft and GitHub integrating more security into, uh, the, the, uh, environment where de developers live every day. Um, and and I think that's super, uh, a super encouraging evolvement of the, uh, of the landscape for developers. More is available to them that they can codify in their development and build pipelines than has ever been made available, uh, you know, before, Which is a good, bad thing.

It's a very good, bad thing. Go ahead, please jump in. I wanted to fill out everybody's bingo card and just go ahead and throw in zero trust because I think, I think this is real though, you guys, I think the zero trust architecture and the programs are working towards, from an architecture standpoint will push us towards the type of communication and collaboration.

And Tracy, I've been sitting here thinking about what you said because I never really thought about it, but yes, from the infrastructure and security architecture standpoint, I guess it has been our job in the middle to deal with any of the vulnerabilities of the platforms and the applications. You know, it's like i d s here, WF here. Like it's, it's Oprah all of a sudden.

Like, you get a wf, you get a wf, you get a barrel, and that's what we, we do. But it, it's even within i d s and I P s, you know, I'm kind of like, we've, we've got new cool words for these things now. Um, but fundamentally the same thing.

We're supposed to tune all of our protections and controls for the protect surface that's behind it. And honestly, we don't freaking know what's behind it. We just kind of try stuff and make sure that when the pen testers come in, they don't get past it.

However we have it configured that day. But as soon as somebody does an update, you know, all of that, we're, we are rebuilding the infrastructure and the architecture each time and it is exhausting and it is not effective. And it sounds like there's some better ways to go about it.

I think that we're gonna sort out better ways to go about it. Um, you know, I'm gonna go ahead and bring up the topic. Uh, AI will help us do better pattern matching and start recognizing problems, uh, early on.

But the core issue that we have on the DevOps side, not so much on the production side cuz you guys have got a lot of data, you can start collecting more and more data. You have, you've been collecting data, you guys have some really good stats and good metrics. But overall, if you really wanna build a really solid AI system to start addressing some of the security issues, not only on this, on, on both sides of the house, you have to have data.

And I'm preaching this all the time because we talk about, oh, we're gonna have these AI systems. It's like, yeah, where's the data gonna come from? Is it gonna come from GitHub?

GitHub doesn't have that kind of data, is it gonna come from just searching Google? It's just, it's not gonna have that kind of data. So as a community, we have to start building better evidence and a bitter evidence store on both what's been deployed, how it's acting, as well as how it's built and all, and everything in between because it's that data assembled together and aggregated at an organizational level that's gonna allow us to start automating and doing pattern matching and make a more intelligent system.

And we have not, as a community, we are not talking about that as much as we should. Yeah. I'll, I'll jump on the AI topic only because at, at R S A I was having a conversation with, uh, my good friend Shannon Le, uh, and she was saying, look, I'm, I'm playing around with the generative AI stuff from a coding perspective and I'm looking at these different platforms that are making code, uh, available to me.

And the problem with some of those systems is they're using outdated data sources. As Tracy says, it's not just about getting data, but even some of the data sources that they have today in terms of what's good code they're pulling from that are a couple of years old. And when you're pulling code that's a couple of years old, that could be code that has known vulnerabilities in it and the vulnerabilities have since been discovered and corrected.

But the AI systems don't, the data within those don't know that. So it's actually, she, she labeled it as degenerative ai, not generative ai. Um, and I think that you just have to be aware, even going back to, to Tracy's point of what's the quality of the data that we're getting from those systems that is helping developers.

We're just at the beginning of a lot of the generative work, uh, certainly within development and, and having access to that. So things will improve. Uh, but a again, I think the expectations are changing.

Uh, the accessibility of tools, accessibility of data and AI platforms is going to continue to evolve and, and get better over time. But it, it requires investment. I also learned a, a new term as we've gotten more into AI and that conversation of data poisoning where data going into the AI system, which may be inaccurate, um, you know, the example of that was given is, uh, home valuation, right?

With, there's some error in the formula that calculated it and that data goes into an AI system, but it never gets updated. And so now that gets perpetuated into mortgages and, uh, and market analysis and, you know, that data stays persistent. It kinda has a life of its own.

So there's new challenges with how we just manage that data, right? We, as from a DevOps perspective, it is a conversation that now I used to preach about securing your builds. Now I preach about collecting your evidence because they're in the, on the DevOps side and even on the software development side, we're producing data, we're producing logs everywhere.

Um, even on the production side, we're producing logs, but we just let them sit out there. Even in Sbam, um, recently, everybody has heard the Biden administration said, Hey, if you're gonna do work with the government, you gotta give us an sbam. Well, industry's pushing back and saying, okay, we're gonna give you an S bomb, but what are you gonna do with it?

Mm-hmm. Why make it a, a, a, a, a part of a contract when you're just going let it sit on a file system someplace? All of that is the evidence that we should be, uh, bringing up.

And it should come from both sides of the house so that we don't have a degenerative AI process because we're pulling old data. We really don't even have a place to pull old data, to be quite honest. It's just grabbing for something.

I I, I don't disagree with you. That, that government, someone asking like, we're going to require SBOs. And the response being, what are you going to do with it?

It's totally appropriate. That said, on the SBO front, when the Linux Foundation went out and surveyed, you know, a, a large audience of IT security development professionals and said, are you using SBOs O only something like 43%, 45% were even using some form of SBO somewhere in their business. Not, you know, all of these, not all, Not everywhere, just right.

Everywhere, right? But even so, like the, the thing is, if, if I go to any development shop today and say, are you, you know, are you using SBOs? The next question is if, if they say yes, the next question is what are you doing with it?

Because a lot of people are like, Hey, I'm producing SBOs, check the box, Check the box. But then it's like, why, why did you actually produce an SBO m in the first place? So now you have a list of what your software is made of, what are you going to do with that?

And they haven't asked the kind of normal follow on questions, uh, and they're just looking for check the box responses. And that gets you nowhere. You have to understand why the SBO m was important in the first place.

And I think that narrative has really been lost by a lot of people jumping on the, the bandwagon of let's all just say sba. And they're like, there's, there's no meat behind your recommendation. It's cool to see what do I do next?

But also, Mitch, um, just recently, Mitch and I had a conversation on CISO talk with, um, Lisa Bradley, right? I'm terrible with names, actually Lisa's local, so I yep. Should know this.

But, uh, at Dell, and we were talking about SBOs and, and the, she has some strong opinions about it as well. Um, but, but that frustrating feeling of, without any context, back to your point, what are you gonna do with that information and what does that mean? Um, and she was kind of talking about, you know, rolling in a vulnerability exchange to understand the context of what was in the SBOs so that you understand what to do with that information instead of having it in a vacuum.

So I'm curious if your thoughts on that. I'm, I'm gonna step in and defend SBOs. Not that you're attacking it, but this is not course you, Mitch, this is not the complete answer, but, you know, in, in the, in the networking security world, we always think the first step is what do we have that we have to, so we know what to secure, right?

It's all about discovery and what is it we're trying to secure? And that's kinda the equivalent of what at least, I mean, SBO m could be for network network software too, but in the software world where we're assembling building software from a lot of different sources, that's what that is. Now the question of what you do with that is the next best question, right?

Um, so that's the stage we're at is, okay, now that we went out and discovered everything, how are we going to use that? And by the way, that SBO changes could be by the minute, by the hour, by the day. It's not a static thing.

That's where I think that's where we need to move that conversation is to O one of the sides of this conversation. It is not r really like whether you do SBOs or not, that's not the question that you should start with. Mm-hmm.

One of the questions you should start with is how are your adversaries approaching attacks to breach your environment and steal your data or get access to your applications? What paths are they taking that are most successful? What paths were they taking that are becoming less successful because of practices or technology or knowledge exchange there?

And if you understand what the attack vectors are on your organization and what the adversaries are doing, then you can determine whether the SBO m or anything else should be the higher priority, right? And, and that, that's really it because if, if, if there's no adversary attack path toward that thing that the SBO M is helping you with, then why invest in it at all? There are other areas that are more, you know, more meaty for the adversaries to go after.

And we have to stop the, so part of our problem is cultural. Um, what we have done when we think about SBOs or if we think about securing a particular point on the production side, our point problems, right? We're looking at a particular container's SBO as one ti it could be a tiny little microservice that has just a few lines of code.

We have an SBAM for that. We have another sbam for another container. We have SBOs everywhere, literally.

And that was one of the, uh, linnux foundation as an sbam ever used to be as called sbam everywhere stream. The problem is that we're not thinking about a broader picture of an organizational security profile. And that is what a collection of SBOs, even as they change a new versions come along, if we're gathering that evidence, we're able to look at our entire organization's security profile.

So we can begin to answer the question that Derek just asked, where, you know, where are our biggest problems in terms of, I don't wanna use the word vulnerability cause it's too much associated with common vulnerabilities and exposure CBEs, but, but where is our, where, where are we? Um, where is the chance of somebody breaking in so to speak? We might have, our production area might be super secure, but we have, our builds are still all scripted, super easy to have nefarious code dropped into them.

Super easy to repeat our log for J issues cuz we don't know what version of it we're using. Again, we have to back up and we have to start looking at an organizational levels profile. And the problem with SBOs is right now, because they're not consumed, they're all point, they're showing a particular state of a particular container or binary at a particular time.

And we, and it's, and we don't use that data to, to aggregate it up to define an organizational profile that we have a nice clean dashboard that we can see everything we need to be, and that's where we need to be bringing this data in. So SMOs are important. Don't get to an idea that you shouldn't generate them, generate them please.

Because then if you need, you have a problem, then you can fix it. But as a, as a community, we need to be thinking about organizational security profiles and not container at a time. The siloed data is a problem.

Tracy, I have a question. I'm curious actually, Tracy and Derek, if we, I, I, I'm a fan of maturity models and I know some of the things I've heard, um, which might be right and might be wrong, I don't know. But some of the things I've heard are that most organizations and developers do have, I don't wanna call it the equivalent of an sbo, but they do have some record of, of, of the different third party, um, you know, code that's, that's in their product usually somewhere, maybe it's not formalized and maybe it's not centrally managed.

Um, so in, in the concept of like something you said, um, Tracy, or maybe it was Derek, was, you know, people can get focused on the wrong thing. And I've heard that several times as it relates to SBO m So my question for you each is if there was like top three or first three things, if we talked about a maturity model for an organization as it relates to secure development practices, what are y your sort of first three foundational things that they should make sure that they're doing? Yeah, uh, I'll, I'll start with one.

Uh, and, and like basic foundational things, uh, if you have an application security team, and if, if you're a development team and you have an application security team, have you spoken with them, right? Just basic do you know who they are? Have you spoken with them and have you done anything to kind of collaborate and working together on, you know, any initiative, right?

And, and you know, that it could be as simple as we have a meet a regular meeting. It could be as simple as we've scheduled a regular monthly luncheon, learn between development and security to share practices. It could be more advanced as to we actually have red teams and the red teams sit side by side or within se you know, similar closely linked teams with our development team to help them understand here's how your adversaries see this application or this infrastructure or this build chain.

Uh, and this is what I would do as an adversary to take advantage of any vulnerabilities or weaknesses within that infrastructure application. So I like the basics come with do you even know each other and are you talking? And I, I don't think, you know, you're gonna get a lot of unanimous yeses out there if you ask that question, but Tracy may disagree with me, so No, I don't, I don't at all.

But I would say, um, to answer the question, I would say start with, make sure you're doing code and repo scanning. Um, number one, that is a simple, uh, pro that's a simple task to, to do. You can do that today.

Um, number two is a little more difficult, and that is make sure that your, uh, DevOps pipeline, uh, is compliant. Uh, and that means that they've spoken to the application security team and the application security team has told them what they want in that DevOps pipeline. Now, the reason why number two is difficult is because we are a group of people who love to script.

And we have thousands, literally, no, we have millions of millions on the production side and the developer side of brittle one-off scripts that are running most of our production environments and running our builds. And you can't build compliance into those scripts. It's impossible.

And even if today an organization like a, you know, a large bank insurance company wanted to generate SBOs for every single project they could, they would have to, some of 'em would have to visit 50,000 plus, uh, you know, scripts, workflow scripts. So they can't do it. It's impossible.

Um, the third thing would be to understand how to get away from scripts. How we can start breaking our habit of everything has to be a script. And certainly if it's gonna be a script on the, on the production side, you know, I GI ops is a good way to start managing it.

You know, production by pull request is a good way to start controlling those scripts, but ultimately moving away from these scripted processes will be, it is absolutely critical in being able to achieve compliance across an organization. Can't do it any other way. I have a Follow on.

You didnt, you didn't ask me. I wanna answer your question. Oh, I can jump in.

Okay. I Alright. I didn't ask you.

That's right. I'm, I'm, I'm gonna, uh, I'm gonna invoke bartender privilege. Get, Get in there bartender.

And then I have a follow up question for cord. Derek. You know, I was thinking about, I mean, what, what you all said, I, I do agree with thinking about what Eric said about talking and Derek said about talking with each other.

I think one of the habits we have to stop is telling ourselves and each other that everything is different. That we're in the cloud now, it's all completely different. Um, we're in cloud native architecture.

That's all completely different because I think that goes to the cultural problem we all experience, which is the silos. Um, organizationally, it's the hanging with the people who are like you. So you don't talk to the other people who aren't like you.

And I think if we start to, to leverage, you know, if you approach a conversation like Derek was suggesting with two ideas in mind, one is what can I learn? I mean, I've got a lot of questions about security that I would really like to know. I'd really like to understand like how does the build process really work?

What do you mean when you say DevOps, workflow pipeline, blah, blah, blah, right? Show me how that talk, explain that to me cuz I would really be curious. I really am interested.

I think the other is, um, there's a lot that we bring from our, our existing domains that still apply just like they did 10 years ago. And one of the ways I like to describe it is if I took all the buzzwords out of cloud native and microservices and all this kind of stuff, what I would, what I would say to a security person is say essentially what we're doing is we're kind of bringing the network into the application itself. That's all those little microservices talked to each other.

And you have things like API gateways and access controls. Same things we do in a network. It's all the same ideas.

We're just now doing it at a much more granular level. Now. Now, now let's talk about how we secure that, or by you, you could have that conversation.

So there's a lot of things that we can do by supporting each other in how we talk as opposed to pushing each other away, which has been our, our natural siloed history, if I will. So you didn't ask, but I answered anyway, so back to you Jacob. And a lovely answer.

It was, uh, go ahead, Tracy. Well, I, I was gonna say the problem again is cultural. We are, um, we always point fingers at each other.

It's our first place to go because failure is not an option. And I believe that failure has to be an option. And failing fast is super critical in allowing teams to be able to have conversations without finger pointing.

It's the only way you can come to quick answers is if everybody feels safe and able to be honest about what happened. Because fail failing is an option. It just has to be a fast failure.

I love everything around chaos engineering. I think it's a, uh, a really amazing way to think of shifting from this finger pointing and root cause analysis. Um, while the, while the fire's so burning, everybody's trying to figure out who did it instead of just fix it.

I like, just fix it, move on. Uh, figure out how we g how we got there later. But, uh, failing fast is important in this discussion.

I have a, I've, uh, there's actually what we did wrong when we labeled created that saying is it isn't fail. That isn't an option. Dying in space is not an option.

That's what's not an option. Failure gifts required, you have to have, they would've never gotten Apollo 13 back without all the failed ideas before they got the one that weren't so exactly, no dying in space. So, uh, I'll, I'll build off of what Tracy said, uh, uh, about culture.

Um, one, uh, conversation I was having with a CISO a little while ago, um, focused around the relationship between security and development and understanding what their needs are, how they work, what kind of information they need when, and one of the conversations that he came away with as a CISO in talking to the development team, the head of development was, look, there are tons of security solutions out there and they produce, can produce an enormous amount of information out there. But I had, you know, as an engineering team, what he heard was, we have an expectation that if the security tools are built into our environment and providing information to us, or if security teams are interacting with us, they have to provide a response within five minutes or two minutes or 30 seconds. And it can't just be, Hey, there's something wrong with this code.

You need to go and fix it, but there's something wrong with this code. Here's what it is. Here's the potential fix that's available if there is one.

And not just, Hey, security's here. We looked at your code. There are problems, or we know how to figure out if there are problems in the code.

Let's just run the static analysis on your code and the results are gonna come through in 20 minutes. Uh, and you know, we'll see the response on our dashboards that are built for security people that don't talk the language of developers. And you know, you have to look for the connections in the culture that you're speaking the same language.

You have the similar expectations of what the results are going to be or what information is delivered and what the service level objectives are. If developers are okay with embedding something in their system that might give them helpful information in seconds or a couple of minutes, and AppSec has this idea of, yeah, we can produce, you know, this information, but it's only, you know, once an hour or once a day, uh, or it's built in dashboards that means something to security people but not developers. That there's a cultural mismatch there.

And I think that cultural matching is part A, again, going back to you, you have to have the conversation. You have to meet the people and find common ground and look for solutions that meet common ground. And as a result, the CISO said, look, if I am a vendor coming in and they're pitching something and they're saying, you know, we we're setting up this environment, we're gonna analyze the code, you know, it'll be finished in a half an hour.

He is like, look, we can't even start. If your solution can't produce an answer that helps my developers in under five minutes, I'm not even talking to you. You're not even allowed in the, in the door unless we have that.

And that was part of the cultural understanding that grew because the, you know, security team was talking to the development team and understanding where those mismatches were. Sorry, I'm gonna jump back in with my questions cause Yeah, please do. I I was holding it from earlier, Derek, when I ask you the first kind of what were the first, uh, few things people should be paying attention to if they had a maturity model of secure development?

Yeah. Um, and this kind of relates because you've been talking about the communication between development and, and security. And my question is, you know, a lot of the clients I've worked with over the years have been in and now are man, are manufacturers, um, and a lot of their development team, whether they're in a startup mode or they're, you know, they've been in existence for, for decades, a lot of their development team is offshore.

Um, and there's a, what what feels like to me based on communicating peripherally with these people, um, is that it feels like there's a big disconnect, um, and a lot of weird paths through certain people to get between the teams we're dealing with here with security and the development team leadership, which might be outside the us. So I'm really curious, you know, is, is that anecdotal? Is that model anecdotal, um, to, to my experience, is that very common in development teams?

And then how do you overcome that both from a, a geography and a communication and a and a cultural standpoint? Uh, you know, certainly the, the pain is real. Uh, I, I think e even, you know, maybe post, uh, or because of covid development teams seems, seems like every organization I talk to is says, look, our developers are a hundred percent remote.

Now, they might not have been pre covid, but they certainly are post covid and none of them wanted to come back, uh, in, into the office. Now, there may be organizations that the developers are, are in the offices and are working together. And as you were talking, you know, my my initial reaction was, you know what a plane ticket is, like go buy one and go meet with the people.

Like if you're in this day and age, if you're using distance as an excuse because you haven't gone out to see the people, I mean, I think that's like outsourcing 1 0 1. If you're not meeting the people, if you don't have the relationship formed, if you don't take the time to have these conversations, if the conversations seem so foreign because you only have it once a year, there's something wrong with the cultural dynamic there. It's, it's almost like, you know, when you're the employee at the company and you have to go through those security videos once a year that are like, don't open the phishing email or, you know, like, use two factor authentication and you see it once a year and it's just so painful to go through because it's not ingrained as a cultural expectation.

The way you build cultural expectation and, and norms is the frequency of, you know, not like we need to get together once a week, and I'm gonna beat it over your head, but just like, what is our cultural norm? What is the expectation that we should all have around this and gravitate toward? And I think it's up to leadership in the organizations to set some of that expectation and practice the expectation.

And some of that might be show up in person, get a plane ticket and go. So I, I heard Tracy suck. Why don't buttercup Let this be our last, we're gonna run outta time.

Ok. So go ahead jj. No, I was just gonna say, it sounds like Derek is saying, I mean, so the common sense prevails and sometimes you, you, you suck it up and maybe you can't have the in-office every two week, but you lunch and learn with the teams, but you, you figure it out.

And if you're on weird time zones, you, you just suck it up and you do it maybe at least quarterly. And I, I, I think it, you know, we talked about like the CISO talking to, you know, development, but it's also the development, you know, leads talking to the CISOs insecurity teams, it works both ways. Like what's your world like, what are your challenges, right?

What makes it, you know, life difficult on your side? Uh, but things are like the environment is getting much better out there on, on, you know, on the venture capital side, I see a ton of investment that has gone into making development and security work better together. Uh, giving them common language, common platforms, tooling that actually produces information in a developer language that is security oriented tooling like that those investments have been made are still being made.

And, and I think that is helping to bridge the gap a little more where technology can aid in those things. But the human element side of, again, do you talk to one another is important. Let's have, uh, Tracy, when I hear from you kinda wrap things up, give your perspective on this and, you know, and next thing we'll dedicate a whole conference to this topic.

Well, you know, um, when you were talking about what we used to call follow the sun development and you have developers all, all around the world, which we do, um, our open source community as everywhere, you really have to have a predefined platform for their, for them to interact with. Um, it, it may be the case that application security can get on a plane and go visit those folks, but more importantly, I think right now, uh, is that we really have to start defining a tighter, uh, more accountable DevOps process. It has to evolve to include security steps in it so that the application security team can define those levels of compliance.

And you might even wanna have a somewhat of a compliance scorecard for each of the teams, so you know what their maturity level is in the process. Uh, but developers are gonna code, they're gonna sit there and they're gonna crank out code, and that's what they're being paid to do. Uh, they have an obligation to make sure that they're writing code that's not, doesn't have memory leaks or may have some, uh, bad code or pull in some libraries that we shouldn't be, uh, pulling in.

But at the end of the day, it's gonna be the DevOps pipeline that, that that's gotta be fixed to, uh, to help, uh, solve this problem. It can't just be left up to the developers. It has to be automated.

There has to be clear levels of, of compliance and maybe a compliance scorecard. So you know where everybody's at is a good idea. Well, the bad news is it's time for last call.

The good news is last call's on the house. Thanks. Thanks to all of you.

I hope we get a chance to do that together, uh, sometime soon. Thank you, uh, jj, Jennifer Manila and Derek Weeks and Tracy Reagan. Um, you know, it's a privilege to be part of this conversation with you and I appreciate you doing this and I know our audience does too.

So thank you audience for joining us and, uh, enjoy, enjoy the other great sessions that are happening today. So we hope to see you again soon. Thank you.

Thank you very much.