Episode 3: The Software Sweet Spot – Balancing Security and Development Speed – ActiveState Podcast

Hey, this is the Active State Podcast, episode three, the Sweet Spot, balancing Security and Development Speed. I'm Nicole Schwartz, the senior Security product Manager here. Hi, and I'm Martin Drohmann.

I'm a tech lead at the extensibility team at Active State. And today we're just gonna be having a friendly argument back and forth about all of the tensions involved in balancing security with the speed of development. To quickly let you know about active state, it's a open source management platform that makes it more like a utility for you to consume all your open source securely.

And we have been involved in the open source community for 20 years and we give our product away to many maintainers. So if you are a maintainer, feel free to reach out to us and we'd be, uh, happy to give you a free version. I guess I'll start with what does security want as a security person, if you've heard of meantime to remediation that the time between when a vulnerability gets reported and resolved, we want to keep that minimal and we've been doing that by pushing security left ideally so it doesn't get to prod.

This means that I've made people like Martin have to start dealing with security scans and dealing with those vulnerabilities along with some of the new hotness. If you've heard about it, Basa, where you get those attestations and SBOs that we know what it is. But as a result of that, Martin, what's kind of that done to your schedule of how you're delivering things?

I mean, we don't meet that often to review our security vulnerabilities. And I feel like it's structured. It's just a new thing because in companies that I worked before, that usually wasn't the case that we did this.

And I think it makes sense, right? Because for most companies, there's the risk of yes, a successful product can be attacked somehow and that is disastrous, but there's also the risk that you don't deliver something that actually makes your product successful. And that's usually what most companies worried about the most.

And so as a developer, I always have to keep that in mind that we can actually deliver the things that have to be done. And that is, that usually means that security comes next. So yeah, it's about the priorities.

You've only got so many hours in a day and maintaining everything and patching it versus being asked to hit certain deadline and striking that particular balance. Yes, exactly. And yeah, as a developer of course, that's also not something that I am necessarily excited about.

Just updating my dependency without, it doesn't change anything functionally, right? I'm more excited about creating something new or improving the user's experience where as security is something more defensive. However, updating dependencies regularly also comes with a benefit because it makes it easier when you actually have to update your dependencies once it is, it is a necessity to implement a new feature.

Yeah, because there's like performance improvement, perform improve, there's new functionality if you're kind of keeping up with the community. Now, from my point of view, what can be interesting is when I look at a vulnerability, I see that it's critical and I see that there's all these terrible things that could happen as a result, but a lot of times I'm not necessarily understanding how that vulnerability fits in the ecosystem. And so I'm approaching it just from, it says it's critical and it says these terrible things, but when you get the opportunity to look at it, you might be able to say, we don't call that function, or, that's not really an accessible piece of our ecosystem.

It's kinda like you were saying things could be attacked, but if that piece is not reachable from outside or not getting user input, then it's gonna be much harder for it to get some kind of overflow error. And so sometimes just being able to work together to really prioritize not just between product features and security, but of those security bugs, which ones really are the most important to patch? And I think right now it's kind of interesting 'cause a lot of security people don't get trained in development.

And at least with the age that both of us are, I think a lot of development classes didn't include a ton of security stuff. So everyone's kind of having to come together and, and chat about all of these issues to see which ones really matter. Yeah, right.

And I, I mean with, to me it's also a matter of, of tooling that you need for this because I mean there, there's always been tools around that tell me what kind of vulnerabilities are in a certain software that I'm working on, but maybe I looked at it once or twice and just got overwhelmed or, or read so and thought, I don't think that matters. And it, it doesn't ever matter. So yeah, I really, it's, I think it's a matter of I I improving the tooling so you can actually reduce the noise and be able to do it because the developer with the current tools gets too overwhelmed if you would react to every, everything there.

I mean, I can't really tell it because I've figured what those tools for so long now as it just annoyed me, but I don't think it, it would do the company good. So something needs to happen there. And I think we are on a good path honestly with how we deal with the active state platform by just making it at first easier to connect it to the ingredients that you have in your open source.

But I think we are also going a little further. Yeah, we're looking at today our enterprise customers, our BE team looks at a lot of the CBEs and then puts comments in there and says, the team's not relevant, you don't need to worry about it. And doing that for our customers, like you said, reduces the noise that they have to deal with.

And I think we're looking at ways to kind of augment that and make that faster and more scalable so that we can kind of do that for everything is, does this really matter? As an example, when I am triaging things, sometimes it'll say this only applies if you're using it in a container that's Linux based. Well, if the customer's using it in a Windows based system, then they're not gonna be impacted.

Then we can immediately say not applicable. So figuring out how to help customers, like you said, reduce that noise. Now you were talking about how if you can keep up to date that you can get benefits of new features, new functionality.

What are some of the things that make it difficult other than just prioritization to keep things updated? Well, whenever you open, you develop with open source software, you're not developing by yourself, right? They're open source softwares.

That's the benefit of it. It it is developed by hundreds, thousands, millions of developers. But it's also what causes dependency hell.

Because when I have my, my my, my application working on Monday and I update my dependencies on Wednesday there, there could have been hundreds, thousands of new additions to the open source dependencies and I will pull them all in on Wednesday. There are of course structures in place to, to mitigate the impact or describes them behind version numbers, but that doesn't always work properly because it's just too big of a problem depending on how complicated your dependency tree really is. A little bit.

But it, but some, and A lot of people have a lot of dependencies, so that can be a pretty complex tree. And also I think packages become more complicated. Some of them really depend on multiple layers of open source software.

If you think of AI applications or, or just data mining that there's so much going in different languages too. So there's always something going on. And the bigger the penry, the greater the impact it.

I mean, if you have a small dependency tree, you might have 10 updates in the open source community between Monday and Wednesday. If you have a bigger one, it could could be a hundred thousand in that time. And yeah, of course it's impossible.

Then if something breaks too quickly, say what caused it? Was it on my side what I did on Mon between Monday and Wednesday, it, was it something the code, was it that, yeah, that it broke and introduced a new feature? Maybe I just have to change a function name or something, but I always have to figure this out.

And sometimes also functionality changes underneath in ways that the open source developers didn't predict because it was third level update that they didn't, that they themselves didn't know about. Right. So they have the same problem actually, that when you're on the, if the change happened layers above what they are developing, they are, they can also break.

I think the saying that I usually hear a lot is turtles all the way down because they're relying on open source and whom they're relying on is also relying on open source. It's just turtles all the way down. Yes.

Yeah. So, and that's what I call dependency. Now, our platform, at least for Python and peril and Tickle and Ruby right now helps people avoid dependency.

Hell. And the UX team has recently introduced that new report that shows you what changed in your tree. So hopefully that helpful to people in that situation.

We're hoping to expand to more languages soon, but yeah, that we found it an issue, so we're hoping to make it more visible and no more unknowns of what changed though. Hopefully that saves people some time. Yeah.

Yeah. I mean with current tools, it's really difficult to actually see which parts have changed. And then even if you figure this out, you might still have to search the internet, find the change logs.

I've ended up, yeah, searching until I, uh, find the, the actual pull request that caused my, my problem and commenting on it. I think there's a push towards we have to update more, which There definitely is with all of the ride in attacks on the supply chain. I definitely think all security teams are kind of pushing to get updates more frequently now.

Yes. But there's a sweet spot that you have to find and that will be tricky and I hope active state can help with that. I think the interesting thing you're talking about is that it can be very frustrating to have to worry about these updates and prioritize that in with your other work.

If the curity was able to help you reduce the noise and be a little more clear about what the updates were, do you think that would help you argue with prioritization to maybe get a little more time set aside because you could say, Hey, we know these things are critical. So do you think that type of information would help you kind of argue for more frequent patching instead of having to wait till it gets to the point that you have to do the patching? Probably yes.

Yeah, I, I do think so. I mean, I I must admit in the companies that I worked before, it was never really a push for security because it was more, it was more in the startup phase. And in the startup phase, you yeah, as I described before, you don't care so much about that, your attacks to your product that you still have to develop, right?

Um, because yeah, what what is the danger of attacking a product that you don't sell yet for you? Not very much. Maybe you can lose, uh, intellectual property this way.

Um, but even that shouldn't be the whole company's focus, right? But I think in the future with the right tools and maybe some best practices that evolve from there, this will become just more of a standard that hopefully will also improve the update process in general. Because I, I do believe with what we found here at ActiveState, we've already discovered a few things that really make updating when you have to do it for a feature also much easier.

Easier because you, you face the same problems of Dependency Hill and therefore some teams even avoid this and would estimate the complexity of a story that involves updating a, a feature much higher than, than it should be, I think because, Because they know they're gonna get into potentially having those upstream changes that might bite them. Yeah, I mean, having to change the whole code base, maybe all tests it needs somewhere new function call and, and then you need to have, be extra scrutinous about that. Everything is still working after that.

Doing this more often I think would really help having more visibility in what actually happened in my dependencies, if you can get more towards this as an, as an estimate of how complicated the story actually is. So if you can make more informed decisions there, just seeing, oh, we would need this package. What would actually come in there?

And can we maybe even mitigate the effects of downstream dependencies by, by not updating them, just keeping it more in the state that we are right now. That would be really helpful. And also, you said something about in some startups there's not much to protect and I think sometimes people don't threat model particularly well and threat modeling's basically where you say, well, this contain like customer information versus the system doesn't, and it's much more important to fix those vulnerabilities in the system that has customer information or financial information than it necessarily is in the other systems.

So I think if we had to kind of summarize what you said, it's, Hey, let's figure out what systems matter. Let's try and update them. Let's see what that update's gonna contain so that it's not necessarily as unknown and, and Gary maybe minimize that where possible.

And then security right now hopefully is our main people with tools like SVA and active state and other systems so that it's not just about ves, it's about which systems are squishy, what risks are there, have we hardened the container? There's all these elements. So mostly working hand in hand.

You can make it so that you've put layers of security in. It's not just about, well, did I write good code fast, didn't yell about, or do I have updated dependencies that the FDA isn't, you know, yelling about it. Well, do I also have it on the hardened container or did I write it so that my tests are saying?

And that's another thing is you've got all these tests, so that'll help you tell you if you can update fast or not. But are all the tests also telling me that I'm not letting users put in things that could poison the system and cause a buffer overflow, so, Right. Yeah.

Yeah. I think that's a separate problem. Don't take data that's you don't need and don't keep data you don't need as the Q person.

That's the two things. Don't take it if you don't need it and, uh, don't store it if you don't need it. Right.

Yeah. Yeah. This certainly needs a different kind of review and it probably is even important in startups.

I think that's something that developers are more educated about though already, I believe over the time, uh, Once over the last decades, I think once you've gotten yelled at enough times by the SaaS tool, you kinda learn it whether you want to or not. Yeah. And it's, I mean, we all use cloud services and the best practices in those documentations are, are getting better, better communicated, better documented, all these things.

So it, it is becoming more of an awareness. It doesn't prevent supply chain attacks though. Yeah.

But luckily, which at least active seen our get current stay current, we're hoping to help everyone see what's in their supply chain, keep on top of those updates and like I said, have an impact report so you can know exactly what's gonna change, which as you said, then you have that next step of, okay, let's make sure the unit has passed and see whatever else a, a result of that. But at least that hopefully makes it a little bit easier, avoids some of that dependency held. Is there any other things you kind of want people to take away or, or think about this balancing?

No, I I think it's really, yeah, the, the balancing of not wanting to go into dependency held, being overwhelmed by the information about vulnerabilities, not being able to, um, to grasp the depth of it and the applicability to my own application versus the, the threat model, right. What comes out of them. So all of the, the things that I said on the first side are really on the first, at first, all of the things that I said at first were really, yeah, is hourly cumbersome.

I think the only way to make them less cumbersome is to do them more often. So that, that's what we always have seen. That was the shift towards it.

It was, it's now easier to deploy because we do it all the time versus deploying just every quarter every year or whatever. And that always became a huge mess. So with the acuity, I think it's more, you maybe plan to do it every quarter, but then you, maybe you don't do it at all.

So the only way to bring this in, I think, is to do it more often so that it becomes less of a pain. And then like you said, it's it, you build that muscle memory as well. So it's not only while wear and less intimidating, but you've got that practice that's just a habit.

So, yeah. com. We've got the free tier for anyone who wants to just avoid dependency hell and have great portable developer runtimes.

If you're interested in our get current day current, well, we'll actually proactively let you know, hey, there's a recommended bunch of updates that you should do. So it's some small bite size pieces, kind of like we were just talking about. You can check out the Get current date, current page, and again, as we cut, if you do maintain an open source project, reach out.

We'd love to give you enterprise version of our product for free. Thanks. Yeah.

Conversation.