Episode 2: Tech Debt or Security? What Makes you Migrate? – ActiveState Podcast

Hello and welcome to the Active State Podcast. My name is Shane Ward, I'm director of engineering at Active State. And I'm Ed Smith, I'm director of product here at Active State.

This is podcast number two, tech debt or security. What makes you migrate, Shane? So you're got the job title and I more or less associate with deciding when we migrate between those.

Let's start with how do you, how would you define tech then? And what do you kind of think of as security? Oh, those are great questions.

Let's start with security, which is really, uh, a balance between how do I know that nobody can do anything I don't intend to do with our systems, and how do I make sure the people I want to do things can do the things they want? So there's that tension and balance there. I think that's important to keep this in mind.

How do I define tech debt? That's tricky because it's kind of a controversial thing. If you look at when tech debt was originally described, I think by Ward Cunningham on the old patent pat, or sorry, Portland Pattern repository Wiki, the C two Wiki, he was really talking about any shortcuts we make in the business requirements to get a product out the door sooner.

And by out the door, I mean we've got two weeks or one sprint or one week, we wanna show a demo to our customer. What shortcuts can we take? What pieces do we not have to address?

What edge cases can we ignore just to make sure that we can put this code in front of people so they can use it quickly. That was the original intent of tech debt, you know, or very focused investment. I think these days, 20, 25 years later, a lot of people see tech debt as anything in the system that I would do differently now.

And this is not a bad thing, it's just a different thing. But I think it's important to keep in mind the distinction between stuff I would change if I were to change everything. Now, if I were to start over now from stuff where we deliberately took shortcuts in addressing what users want in order to get something to them sooner.

So if we set those up in mind, that's how I personally define tech debt. That's why I personally define security. Does that resonate at all with you?

Yeah, I, I think the interesting thing about you, you mentioned the, the definition of tech debt has kind of shifted over time. What's interesting about the new definition is it's subjective. Oh, I would've done that differently.

Well, I, I don't know anyone who does everything right the first time and I hope we're all kind of learning to be better. So every time you do something, by the time you're done, you should look at it and go, ah, I I could've done that better if I just had more time. So does that kind of imply that everything we do is naturally turning to deck debt or do you feel like, oh no, is, is there a definition of the work that doesn't just automatically become tech debt or we ever doing anything that's good enough?

Probably. I mean, if you look at some of the work like NASA has done or things like that where like the documentary, like they write the right stuff. If you know, for example, you're going to send a probe into space and you have nobody out there who can swap parks.

If you've got cosmic grays coming in and things like that, you know that you have a launch window of this week and otherwise you're going to miss this and you're going to orbit maybe something hopefully for years before you get where you want, you know, all these constraints up front and you can design and implement your systems to those constraints and realize that those aren't going to change because you are not going to change orbital mechanics. Most of the projects we do aren't like that. You know, some of the products we're building at Active State right now are like, we think our customers are going to like this, but we're going to take something and put it in front of them and say, how does this fit with you and what you want to do?

We're gonna get feedback from them and come back and improve these things. So we have a deliberate agile process of saying we're not going to overinvest in building something we think people are going to want. It's more important that we are able to get feedback from them hands-on use of an actual project, an actual user experience, and then we'll change how we think about building the next phase.

So we, we are deliberately, I think taking on some of that original Cunningham definition of tech debt saying we think these things might be wrong or we're willing to accept being wrong because it helps us move more quickly. It helps us not overinvest in things that turn out to be at and best the wrong activities and at worst potentially useless or actually harmful. So yeah, I think we naturally generate this type of technical debt just as a matter of process.

And you mentioned something important there. So the, the customer might see this in a completely different way. So from the, the artist, the coder, the person actually writing the, the program, they see, oh, I could have done that better.

The customer doesn't see the same thing, right? Like they see, right. If anything they experience it as slowness or, oh, that's odd, but rarely do they go, oh, this thing's full of tech debt.

Like any product, I don't think anyone looks at a car and goes, oh, this is full of tech debt. They go, oh, this is complicated. Oh, this doesn't work very well.

Oh this is slow. So that's right. Part of just on on the my side of things where I see it is not, oh this, we could have done this better because I think everyone as a developer should be naturally feeling that way about their work if they're getting better, the question of, oh, when does this actually impact customers and how do we make those decisions of, okay, well we've got this thing that we know we could have done better and we're always feeling that way, but what's the time to actually resolve these things?

What are the important things to resolve as far as technical debt goes? I think that's really the, a bit of the balance here. And just coming back to the central question of when is it time to migrate?

I think if you ask a person as soon as they're done, oh, could I have done this better? Is that time to migrate? Probably not.

And once it starts impacting customers, once you see performance issues, that's that time to migrate maybe. And like where do we draw that line? Because everyone thinks things could be faster.

In a perfect world, it's when does the impact felt? When does it impact, you know, sales and things like that. The actual things with what we do kind of becomes a question of tech debt there and we think we can make this more concrete, right?

I mean, I could probably upset a lot of people by saying the formatting of your code doesn't matter. I don't believe that, but I could be polemic and say that. But I think it's more useful to say there is duplication in this system.

The customer doesn't notice that these variables are named the wrong thing. The customer doesn't notice that this code might be difficult to modify or difficult to test. The customer doesn't notice that until the customer does when it affects the quality of the software, the reliability, the performance, things like that.

But those are things we can measure in a way that is less subjective than I'm looking at this code and I feel nervous trying to end it because I feel like it's fragile or I wouldn't have written this way. Or I think there's duplication here. And so it, it's important to me at least when I'm thinking about how to address all of these problems in an engineering team, be able to say it's worth investing time and effort fixing these problems, these problems they're not worth investing in right now compared to the other.

Right. Because I know as an engineer myself, I would love to have all the time and effort and ability in the world to make everything polished and perfect for whatever subjective means that is. But what I thought was perfect when I started, it might not be perfect when I get done.

And that may not change how the users of the software perceive it or how it meets their needs or how it, uh, produces business value there. So that to me is important distinction to be able to make. Yeah.

And the, the phrase that gets used, a lot of it ain't broke, don't fix it. And Right. I think as people who are interested in doing better or running better code, right?

Eight broke is a pretty broad category, right? There's stuff that works, it's entirely subjective, it it can work while it's on fire and it's working and people don't see it as long it's under the covers. So when do you think it's time to actually change things over when you, you've accumulated enough tick tech debt, like, or what are kind of the things you look for in a project or a thing or you're reopening it, you've inherited something.

What are the signs that you go, oh, it's time to do something significant about this? To me, it's all about change in the cost of change. You know?

So for example, if I have, I remember this from like the year 2000. 1 computer that was running an oscilloscope in a lab. And at some point that computer is going to stop working, capacitor is going to blow.

We're gonna have to replace the motherboard. That oscilloscope is going to be down because that processor, that machine is going to reach the end of its hardware life. There is a point in time where we're going to have to make a hard decision or that decision's gonna be made for us.

We need to replace this. So there is I think a natural endpoint to software and systems we need to keep in mind. That's one piece of it.

The other piece of it is really along the lines of it, it broke. Don't fix it. If I don't need to modify something, it doesn't really matter if it's hard to modify, it only matters when I need to modify it.

You know, if I'm worried about this particular piece of code, this particular function, this particular object, this particular library changing out from under me or being fragile or, or not working very well. When someone goes in and makes a change, it always has bugs elsewhere. If we never touch that, it's not ideal, but it is a trade off we should consider making because if it is working and stable, at least for what it's doing until someone modifies it, we can put off modifying until it's necessary.

But again, that choice may be made for us. Similarly, you might say that we're using um, Cintas six and it's been working fine for us. That has an end date on it.

Python two seven has an end date on it and it's going to get more and more expensive, more and more difficult to move away from that because there are going to be fewer and fewer people. There's less and less support for keeping that thing up and running. So sometimes we can get out of that.

Sometimes we can't. My personal preference is to say anything that we can delay because it's not the most important thing we should delay. I don't wanna modify things just because they're old.

I wanna modify things because you've got a plan to migrate away from them, which is probably answering the question with the question, how do you decide what your migration strategy looks like? Right? And I, I think the way we kinda see that cascade, 'cause it's a perfectly rational choice at the time, right?

Yeah. These are all perfectly rational choices. I think we should assume that we should assume been fakes.

Yeah. And then as kind of time goes on, you go, okay, well let's prop this up again. Let's not look to this.

Let's build another thing we could have done better around this to kinda keep it going. And then eventually you've got this structure that's made out of one-off decisions and rational choices that suddenly very hard to break in. You, you kind of mentioned the issue of uh, the difficulties and the upgrade.

The difficulty is like actually making a change with systems that are complicated and the more layers you add onto it, all of a sudden you've got this armored onion of things that you have to get through just to do an upgrade. And you go, well, it'd be easier just to attack another layer onto this than it would be to upgrade. But I think this is the point where there's a certain anxiety that starts to grow and oh, we should have done something about that 10 years ago.

Oh I didn't, we, and then if you went back in a time machine, you'd be, oh, there's tons of other priorities not fixing this thing. And then 10 years later, here we are. And that, that's where I find people generally get to.

It's never anyone goes, oh, I wanna stay on the oldest thing forever. It's just they find themselves there eventually and they realized, oh we've reached this breaking point. It's very hard to isolate 'cause it's not one big decision or one big event.

It's suddenly, oh, that's too old, we can't do anything about it anymore. And all the things we've done, the the guy who knew how to tweak this is retired. There's usually some other event that brings up the choice is made for it.

Yes. Yeah. Since either retirement or the hardware, you kinda mentioned some of the, the older stuff two seven sent to s sometimes it's, oh this is the last guy, the company knew how to do this and you're relying on him and he's won the lottery and gone off to The Bahamas.

Now what do we do? So that's that the generally the, the choice is kind of made for you. I think there's very few companies over the luxury of being very proactive about this often because, oh, the system works for profiting off it.

Don't look at it too hard. So in those cases, it's the situation that the choice gets made for you. And I think it's similar but a little different in the the security space, right?

Because sometimes there's mm-hmm, it's an outside influence again, but sort of an external factor rather than oh no, this thing has failed. It's oh, there's a problem. Oh there's a problem that's super significant.

How do you kind of see those things now just not a security issue, uh, pushes you to make a change? And we've already talked about it's the same sort of systems, right? Like it doesn't make it any easier whether there's a security issue, it's just a layer on top.

And I think it's also unpredictable, right? Uh, I think one of the difficult pieces of security is you might be running this piece of software and it has a vulnerability. That vulnerability was there.

Since you deployed it, it's only after someone has discovered it or someone has exploited it that you realize we were not safe all along. So that decision gets made for you there and it's not necessarily the case that keeping up to date always protects you from that. To me, the question is really how long does it take me after discovery of a thing to make sure that I am no longer vulnerable?

Some of that is going to be your other security practices. If you're practicing defense in depth, for example, if you're practicing at monitoring and auditing and logging and things like that, you have much, much better capabilities to react to something untoward like that. But part of it is just going to have to be, you need to acknowledge there are going to be forces outside of your control you are going to have to react to.

And your ability to react to those as quickly as possible, as is reasonable, is going to be your ability to protect yourself. I think in those cases. And which of the, the, so teched versus security, which do you find kind of carries the bigger stick as far as motivating people to make those changes?

Because often tech, tech comes from an internal, okay, now it's, it's finally time and security often comes from an outside influencer even, oh the, a new VP started and he's got a security mandate. It's time to make these changes. Right?

Which do you find one gets the company under mo more momentum to make a change? Or do you find it's sort of like equally weighted? That's a great question.

I mean, you could say legitimately that they are both existential risks to a company. This potential, not this potentiometer, I'm sorry, we'll edit that out. This oscilloscope hardware goes down all of a sudden we can't do verification in our lab that could cost us millions of dollars if we got a tape out coming up for example.

Whereas if something like Heartbleed or Log four J or Shell four J comes out, you are going to spend how much time, how many days wandering around all your systems trying to find do we use this? Where do we use it? How can we get rid of it?

What's our migration plan? Right? You have all these potential outside, external shocks to your system that could come up.

What is your capacity to react to those and absorb those into your schedule and your resources and your planning. With that all said, even though they're both existential risks, I think security probably has the more visibility. And so just in terms of, are more people going to ask you about it?

That's probably the case with, I haven't seen a lot of security questionnaires that say, are you using Cintas six? Are you using Python two seven, are you using Pearl five eight? I have seen a lot of security questionnaires that say, do you keep your systems patched?

Do you keep your systems up to date? Have you did internal penetration and testing? Do you have disaster recovery procedures and things like that?

That may be the case that you find bugs where you shine a flashlight in the dark. I suspect it might be, but we should deal with the world as it is. That's where people are shining the flashlight in the dark right now, probably more on security than they are on technical debt, on aging systems, on on delayed migrations.

Right? So I think you're saying it's more of a process of how do you stay on top of these things? Like one of the factors that's gonna boil over someone carry a little bit more weight than the others, but Right.

What, and I guess that leads me to like, how do you stay on top of these things in the real world? Because if you're just waiting for the threats to come to you, if you're just waiting for something to break, that's very react. Is, is there a way to be proactive and on top of this and reduce that anxiety over time or, or know when it's a good time to make these switches?

I think you have to make it a priority, right? Things you don't do very often, you're not gonna be good at. And if you say, our upgrade policy is we're going to upgrade once every two years on Memorial Day and the entire team is going to come in over the weekend and work, what you're saying is, this is a really big bang event.

We're gonna bring in pizza, we're gonna bring in donuts, we're gonna bring in mocktails, we're gonna make a thing of this. But this is an extraordinary circumstance, an extraordinary event. I think what you're saying in that sense is we are not going to get good at doing this.

Whereas if you said, alright, every month we're going to take a service and say, if it's more than six months since we made an update to it, we are going to test out upgradings dependencies and redeploying it. We're gonna test migrating it to a new system. That's going to be painful, that's going to be awkward at first.

But anything you do in a repeated fashion like that, a consistent fashion, you're showing me you value doing that and you're going to lessen the effect and the cost of making these migrations, making these changes, making these updates because you have to, you're forcing yourself to do it. I think one of the reasons people don't upgrade their systems is because they're worried it's going to be expensive. They don't know the scope of it, they don't know the di difficulty of it.

And those unknowns weigh heavily to say, I don't know what benefit I'm gonna get from this. I don't pay for it. These other things I could be doing.

I know how to estimate the effort that I know how to estimate the payoff from that. But what we're looking at here, oh, go ahead. I I was just gonna say, I think we see that model with stuff we actually use on a daily basis, right?

What you, you, you think back to, oh, it used to be a, I would buy a new computer with a new operating system every couple of years, and that's how I wouldn't actually upgrade. I'd just switch my system now. But now our, our cell phones, our watches, anything that's digital, oh, it's time to update.

Oh, can I update tonight for you? Like, it makes that part of the practice instead of waiting for there to be a vulnerability that reacts to often they're a little bit ahead of it or they're patching something that could have later turned into a critical vulnerability, right? Just because, oh, every couple of days we're gonna send an update.

It's not gonna be a big thing. We're gonna make these incremental changes. We know we should be making, make them on a regular basis.

Hey, that acceptable part of the practice. Like, I don't think we even think about those updates or you just see it go, okay, I acknowledge that. Versus in practice, when you're working on a big pace of technology, often you assume it's gonna, oh yeah, this will be good for a little while.

You mentioned once a year doing an update and it feels like the need to do those updates is accelerating unless you're ignoring them. In which case, oh, you're making the problem worse, right? We're not getting a big envelope full of floppy discs in the mail.

Now we can do over the year upgrades for a lot of our systems. I know a lot of people listening to this have air gap environments. They don't have quite that easy path, but this is still something I think you can plan for and execute on.

And I think anything you want to make a repeatable, scalable process, you can do that and you can understand the costs and the benefits of doing this. You're going to have to get good at this if you're gonna do this on a regular basis. And I think that's the only way to get yourself out of this situation is to make this, uh, a capacity of your system.

Think about we are always going to be migrating our system and that may be shutting down old systems. We don't need, it had a start date, it has a maintenance date, it has an end date, it's gonna be replaced for something new. It could be just routine maintenance on a system.

I remember at, at one point, this is way back when the, uh, consultant and I said, we're going to have to reboot this old windows mt box we have in the corner because we can maintain this, we can monitor it, we can bring it back up. If it fails over a long weekend, we're not available. This entire department is going to struggle.

While it's not there, we're choosing to take on this potentially difficult task now because we've made the resources and the time to address any problems we have and, and deliberately fix them rather than being under a really hard deadline to fix them when some external factor came in at the least convenient time. Awesome. So I think what we're coming down to is it's not one reason or the other.

It's if you're not, if you don't have a plan to keep up to date, you're going to fall behind and one of these things will make the choice for you. Is that kind of a, a decent summation? Well, Evan, I appreciate the chat.

Always good to talk to you. I think there's more to say on this, but let's leave it here for now. Sounds good.

Appreciate you taking the time to talk with me here, Shane. I hope everyone enjoyed that. And thanks for listening to the Act of State podcast.

There'll be many more episodes coming down the line here in the near future.