Letting Go – Techstrong Research Review EP 23

Hi everybody. Mike Rothman here, general manager of Techstrong Research with another episode of the Techstrong Research Review. This is our pseudo weekly.

We like to do it weekly, but, you know, life and all that other stuff. So we do it mostly weekly research meeting where Mitch and I talk about, you know, things that we're coming across and, and then, you know, just bounce some ideas around and, and, and really kind of try to flesh out a lot of, you know, what we kind of run into on, on a weekly basis. And mm-hmm.

With that, I probably should introduce my partner, Mitch. Ashley, how are you today? I'm doing great, Mike.

Uh, thank you much. I'm c t o with techstrong and working with Mike is principal, uh, analyst with Textron Research is, I'm excited about this topic. I can't wait for you to get into it, so please, Let's go, let's, let's just jump right in.

Let's jump right in. Yeah, so it was, I had an interesting conversation with, with an old friend of mine, and they are early in the whole, you know, kind of agile and, and DevOps journey. This is very, you know, synchronous with the, the work we just did with DevOps on RAMP and really helping folks really start to think about how do I get going on this journey?

We tend to stick in our own echo chamber, you know, relative to the folks that are, you know, on the front end of DevOps on the front end of cloud native, you know, very advanced from a, you know, kind of programming security or securities code, uh, type of, of concept. But we have to remember, it's a big world and it's a, it's, it's associated or aligned on a bell curve. And there are still lots and lots and lots of folks that are trying to make sense of what this DevOps thing is.

And, you know, once you start to get going, you know, you tend to run into a little bit of an issue, which is you want the developers to do what, and, and you've kind of embraced infrastructure as code and you've built out some of these templates. But you know what, you have a pretty stringent pipeline for, you know, how you wanna make infrastructure changes. And if you think about it in security lingo, it's Im, or security group, or network access, you know, type things.

There are key access to keys and key management types of, of policies and rules that you have in place, right? And, and you make, you know, them request a change through a pull request in a pipeline, and that's great. And then you have, you know, the ability for the operations folks to go in and, and, and review and make sure that it's okay.

And then you kind of run it through the pipeline and it goes through, right? It's the review part that, you know, you tend to start to get into some friction, right? You know, because again, and, and I know it from a security standpoint, obviously Mitch, you know it from, from a lot of the operational standpoint, I don't think they're that different philosophically, but these folks feel responsible for the environment.

And there's fear, right? There's resistance, there's reluctance to let the developers make changes, have that run through a dev environment, and then maybe start to figure out how and what those changes do in a test. Can we get it in dev?

Maybe if it's an obvious thing. Where's the actual review? How many humans do we have to have involved, uh, in this thing?

I, I kind of hold call the whole concept, how do you let go, right? How, how, how do you get to that point of letting go and, and, and accepting the fact that in order to scale, in order to really adopt these DevOps principles, you have to let the developers start to do stuff. And there's going to be things that are spilled, right?

There's going to be mistakes that are made, and it's really a mental thing that folks have to get through. And I had a couple of suggestions for these, officer. I, I want to kind of get your your take on that because I, again, I've got my own opinion for, for what I think the, the, the right answers are.

But, uh, I know you've lived this, so you, you know, I'm sure you have some practical, you know, kind of feedback for, for these folks as well. Well, yeah. And, and I, it kind of had one toe in the security world, both creating products and, and you know, being C G O for security companies, but, you know, definitely more of a, a network and software person, uh, coupled with that.

So I, I think, and unfortunately that helps inform me just understanding kind of what security's about and the people who do it. Um, though I think I understand software and and operations folks a little bit better here. Here's what I think the rub is, is software.

It's kinda like when we, we said we're gonna go to the cloud, you remember, we can't go cuz it's not secure, right? In, in a way we're going through another one of those peaks where, well wait a minute, I'm not gonna, I can't secure things if I don't know what's changed, right? First of all, I need to know what I have, then I need to know what's changed and that I need to have controls, which are processes and, and data and governance, et cetera.

Um, and those structures are, are very good for what they do. What they rub up against is things that are in a continuous loop of change rather than an incremental change. Not to get too philosophical about it, but software has gone through this as well.

Um, cuz even going to Agile, uh, you went from big monolith, you know, first software release I did coming outta college. It was a year before we wrote any code. It was two years before we got the first release out.

Oh my god, mine Mine you by the way. Yeah, same. Now if you don't have something out in two weeks after you start the project, you're like, yeah, You suck.

Sorry, What's matter with you? You suck. You're stupid.

So, you know, even with Agile, we went to, which is a really great thing, we went to a time constrained release. We said two weeks we're gonna release code, whatever we've got, we'll release it provided it meets all safety, security, functionality, testing, et cetera. Um, now what we've done is say, let's put this, take that same thing and just put it into really small loops that happen continuously.

And, and software teams are not all here yet either, right? But the way software gets built, it gets built incrementally and it gets, goes through this continuous integration process and eventually it gets deployed into a test and then at some point it makes it into production. Either somebody approves it or automatically, right?

Less frequently. So that's this continuous cycle that inside of IT software is being changed and created all the time. And so I think what we've gotta do is figure out how do we take what security people do and think about and how they think the best of that and apply it inside something that's fluid, not something that's static.

And that's where that's kind of, hmm, head scratcher, how do I do that? So I want to know things like when does an API change? When do, where are credentials and where are they stored and how do they secured, did that change, um, data, uh, where is it, is it secured and has something changed about how much of it's going somewhere or being used or whatever.

I'm just picking kind of, you know, random things here. But those are the signs that, oh, this is something happening here we would like to know more about. It's kind of like your intrusion detection system telling you, you're certainly getting, you know, back in 2004, suddenly getting, you know, a lot of I love you virus attacks or whatever.

It's those things that we know that are signs that, okay, this is where to focus now. Uh, I, but that, that's kind of my view of it. And, and I think there's a lot of things we can do, but it's getting out of the gatekeeper and things are static mode and getting into the flow mode of things are happening.

I hope I don't insult anybody by saying that cuz I, security people got a crap load of work to do and they work very hard. And I don't mean to, I'm not saying that or they don't at all. So hopefully that makes sense, Mike.

Yeah, It, it does. And, and kind of the way I framed it out, what, what was, you know, and, and obviously there's a multi-pronged approach to, you know, kind of, uh, addressing a, a number of these issues to me, you know, first is spending some time and, and you know, we talked about security champions extensively at DevOps, uh, OnRamp, right? We talked about it at DevOps Connect, right?

DevSecOps is now, or DevOps now DevSecOps, our RSA event that will now be virtual in early June. Um, so you'll be able to see all of those, you know, talks there. We edit, uh, uh, a session on security champion.

So starting to train the developers on, you know, kind of best practice, not just for security, but also operational aspects of that too. Mm-hmm. I dunno if they're operational champions, uh, in the same way.

Uh, but platform engineers I think are starting to, to play, you know, kind of some of those, you know, specific roles and helping, you know, kind of the dev start to work with the foundation and dealing with the stats. So the first area of comfort is to go, you know, I'm not surrounded by idiots, right? And not that you're developers idiots, but they're just untrained, you know, from the standpoint of, um, you know, what these specific issues are.

Uh, and then there's really reinforcement, right? You know, kind of the guardrails around the environment to, you know, kind of intervene before something is overprivileged or intervene before, you know, something is open to the, to the internet to intervene, you know, before, uh, something is expanded that's either gonna blow out the budget or, or, you know, possibly contracted that would impact availability, you know, on an operations side so that you've got, you know, some controls. You know, there are just some things that I never want to happen in that environment.

So I'm gonna put in place guardrails to make sure they never happen. That could be in the form of Azure policies or AW scps or whatever Google's, uh, analogy is. But y you know, again, some, some structure so that things are, are within boundaries, and then you have to have the ability to test, right?

So yes, I want to be able to test the templates, I want to test the code, I want to do that within, you know, kind of the, the, the CI process and make sure that I'm not adding an SCA as I doing software composition analysis is obviously a key part of that to understand where your vulnerabilities are from that perspective. And when you're talking about infrastructure, you're talking about operational things. Again, posture management is critical, right?

And you have to be able to test these, you know, in a test environment. And it was funny, they, these, these folks I was talking you yesterday were talking about, oh, now we're getting pressure from the developers to, to, you know, make these changes in all three of the environments at the same time, right? Jeff, test name problem, like, yikes, right?

Yikes. Because that defeats the purpose. We let the developers do stuff in Dev, we test it and make sure that it's not impacting posture, you know, negatively in test.

And then if everything all checks out, then we move it to prod, right? It's a logical progression. If you break that progression, you know, things are bad with a few exceptions, right?

So then they said, well, you know, we use Circle CI as our, you know, c I CD pipeline. Obviously had that well publicized, you know, kind of security vulnerability, uh, a couple of months ago. Yes.

In that case you have to, you know, kind of make all those changes simultaneously because that's the underlying structure. What you want to do is isolate those changes so you're not pulling along all sorts of application changes at the same time. You're just, you know, kind of changing the underlying substrate, but there's a progression here and, and start slow right?

Quick wins so that, you know, kind of you're in a position where you're able to get comfortable with what's happening, right? And, and, and start to, and then you can, once you gain that comfort, you can start to, you know, gain velocity and start to, you know, let the developers do, do more and more. So it's an incremental process.

But, and again, not, or, or maintaining and, and insisting on manual review for all of these changes. Again, it's just, it's counter to the ethos of DevOps and man, it just doesn't scale. So at some point we all have to let go.

Like, you know, was fortunate you, you know, kind of got a car for, for my kids, you, you know, the other day and they, they've been driving for four years at this point, right? Three 40 years. So mean they're recently experienced drivers, but you know what?

They're in the new car. They're driving back to their house. I'm following on the location, like, you know, to make sure that they got there, you know, because Yeah.

You know, and at some point, uh, that was tough the first day, and then y you know, kind of the next day I followed a little bit and now by the third day it's okay, you know, I've let go. I know y you know, that, that, you know, at least they're, they've got a good chance to, to be okay from that standpoint. And that's where we have to get to, right?

With a lot of these specific discussions. I remember when my, uh, my, my oldest daughter learned how to drive six months after she had her license. I wrote was riding with her.

I was like, damn, you need to slow down. You're gonna like way too fast. Give yourself some breaking room coming up.

Just, just some advice here. Says, I don't know why you drive so fast. She goes, well, that's how fast you drive, dad.

Oh, Okay. There's a lesson there. Important lesson.

You know, what it reminds me of Mike, is, you know, there's some things that carry forward, but you know, who was it, Einstein or whatever said the pro, the thinking that got you to the place of the problem is not the same thing. Right? That'll get you out of it.

I'm not saying it's a problem per se, but it, it is. Some of those things that we do in security will service well and some of them gotta change. And if you really sort of step back and, you know, maybe good time to, maybe we do a little DevOps tutorial sometime for security people or something.

Um, you know, DevOps isn't just this thing developers created, so it's based on manufacturing and the flow of manufacturing and principles like bottlenecks and how you work on bottlenecks, you know, theory of constraints. Um, so, you know, work on the bottleneck cuz anything before it, you just make more things, stack up the bottleneck, work on what happens after, and you just create unused capacity. It's built on things of incremental, continuous improvement like quality, but using the same thing for capabilities.

So those are, so if you kinda lift some of those ideas out and say, well, this is what had to do with my IT group up in 2014, starting DevOps. Like, okay, well let's not worry about DevOps, let's worry about those ideas and how do we start doing that in our work? And we were running security and infrastructure and IT and applications and app dev and you know, we had to start thinking about, well, how would you do it if, and and I've even had this with people since like, uh, don't think about it as a big release thinking about of a number of small releases, um, in this kind of counter-cultural idea as a part of quality.

My view is your speed of delivery people judge quality of what you do by how fast you do it, as well as the quality of it, right? Okay. So if it takes nine months to get it, the whole thing, when they would've waited nine days to get part of it and been just as happy knowing you're gonna get the rest of it, that's, that's quality.

So it's a different kind of way of looking at it. So I, you know, I think that's, it's hard, it's hard to apply some of these things, but once you kind of see it, the flywheel turn slow, it doesn't have to go 8,000 RPMs. You can see if you thought about security of instead of locking thing down and kind of letting it go until we need to fix it.

What if we, what if we, okay, we've done that now think about how we can incrementally improve security. Maybe we make, make bigger steps here and there, but think about getting, make friends with the DevOps tool chain DevOps engineer. Figure out how that flow works.

Where can we put in better secrets management? Where could we bid in, put in, oh, this is a good place for, I need to understand software composition analysis because open source code is now an attack vector for us. You know, get to know that flow and then figure out how you not getting the developers to improve writing better code, getting you to be better at injecting security, building security into the process.

Yep. That's how I think of it. Yeah.

And, and, and, and I do think that there really are, you know, there are two aspects of this, right? I mean, Mitch, you, you've been focusing on a lot of the application layer stuff and I think that's, you know, critical, but security is a little bit tangential to those folks and, and we're trying to insert ourselves into a place that we never really were. When you start thinking about infrastructure as code and the transition, you know, kind of that you make from, you know, actually having someone sit there with a device and, and racket stack it connected and, and configure it, right?

You know, and, and to move towards, you know, this idea of, you know, first virtualization and then, you know, public cloud and then obviously, you know, kind of, um, you know, everything as, as software, uh, on the, on that front, y y you know, so that's the transition where these folks have a lot more historical ownership. Mm-hmm. And, and where I think letting go is, is, you know, critical.

So, so y y you know, they have to let go of, you know, kind of controlling the entire infrastructure. Yeah. Because we have to think about these applications as stacks, right?

Not just code that runs on top of an infrastructure that we, you know, kind of put into a place and then, but, but the reality is the practices that we put in place, whether we're talking about, you know, a software based infrastructure or applications that run on top of it, same stuff, right? Mm-hmm. Same stuff mm-hmm.

Same disciplines you were talking about, um, in a, in a lot of cases, same tool sets, right? And tool chains. Because we are starting to see things like s c a, things like, um, you know, infrastructures, code template scanning that are starting to show up within, you know, kind of not just the posture management, but also, you know, kind of the actual pipeline tools.

So GitHub actions and be able to call some of those tools within that. Obviously GitLab having a very comprehensive, uh, solution along those lines. So that's kind of where we're getting to with mm-hmm.

You know, kind of those things. But I, I, I think the main thing is as we really start to, to wrap up is, um, part of understanding DevOps, part of really embracing this way of building software is the fact that you as, whether you're an operations professional, whether you're a security professional, you have to start to figure out how to let go of having to, you know, turn every knob and make every policy change and review every, you know, uh, little, little thing that, that is a little bit different in the environment. You, you just can't scale that way.

So that's a mental shift that you all have to make. And again, once you're there, you can't even remember the old days when you had to, you know, get involved in everything. But it is difficult for a lot of folks to get there.

And, and, and that resistance is natural. It's understandable, but you have to push through it because you can't get to where you need to be if you can't figure out how to like Go, go. Um, you know, the, you you're talking about software stack, it, it's the whole software stack, not just the app, right?

Because we're talking about Kubernetes and containers and lots of different things that are elements. Go make friends with those. That's the infrastructure to applications.

Go make friends with the people that do that, right? Someone told me is, you know, the way to keep your enemies closer is go make friends with them. I'm not saying those are your enemies, but you know what, you go out, go make friends with those people and start to see how that stuff works.

That's part of this whole securing the infrastructure, right? That's where security people can also play a major role and not have to get into, I don't know how to Ray Python and code. Well, you don't need to know that, right?

There's people that build part of that infrastructure too. So there, there's lots of opportunities. If I get the fear going through it, hello dogs going through it.

There's a lot of opportunities to do that. I think that's, that's the, uh, sign We're at the end of our show. It, it, it, it's, it's so with that, before Mitch gets attacked by his, you know, massive dogs that, uh, that, that he runs herd on most of the time when he's not teching, you know, one thing or another.

We'll let him get back to that. So with that, another episode of Techstrong Research Review, uh, do we have anything coming up? I know we've got, uh, uh, we've got some virtual conference.

Can I Keep cloud native now is coming up Cloud native now? Yes. Cloud native now.

And we're starting to work on DeVos experience. Another one coming Yeah. Dev experience in the fall.

Right? Right, right. I think we have data ops in the, in, you know, over the summer too.

So we have data ops, so lots of things going on, as always with, with Techstrong. So keep an eye on, you know, kind of everything, but, you know, keep interacting with us, uh, at Techstrong Research. We do appreciate everybody that, you know, kind of follows along and, and, and listens to us.

And we'll be back next week with, uh, another episode. Thank you everybody.