Development Environment Security – Tomislav Pericin, ReversingLabs
The weeks-long compromise of CI/CD vendor CircleCI in December sent shockwaves through development organizations around the globe. It also highlighted a key vulnerability facing development organizations: compromised and rogue developer accounts that abuse access to internal networks and assets, and development pipelines. What steps should organizations take to counter this risk? In this expert analysis, ReversingLabs chief software architect Tomislav Peričin unpacks the CircleCI incident, talks about the growing risk of attacks on developers and development pipelines, including secrets, and talks about steps organizations can take to shore up the security of development environments.
Transcript
This is texturing TV. Hey everyone, welcome back to Tech strong TV. I'm really happy to have Thomas love parrot parachute.
Hopefully I got it, right but Thomas left correct me if I'm wrong. I got it cool. Thomas love is with reversing Labs.
He's gonna tell us what he does there in a little bit about himself Thomas love welcome to text drug TV. Thank you. My name is Thomas leperchen.
I'm the chief software architect and one of the founders of reversing Labs. I've been in security for about 20 years this year. We've been doing quite a lot with reversing Labs trying to detect rats.
We're interested in software supply chain security and I'm happy to be in the show. But we're happy to have you on so Thomas Club your arm in security 20 plus years as well. Give her our audience a little background like what kind of roles have you had in security before founding reversing labs?
Now interesting. Well, I was a reverse engineer and I was a reverse engineer at reversing Labs as well. That was one of my you know interests back in the day.
I wanted to reverse engineer malver and all the other things to make sure that you know, so tried detection Solutions have a better chance of detecting threats before they infect the user. Now. I'm kind of extrapolating that knowledge and and using it to protect the software supply chain, which is roughly similar.
I would say but a completely different problem space, you know, compile things versus scripts and so on. Yeah. Absolutely, look, you know I back in the day when?
I was doing more security obviously reverse engineering malware, especially. Was I mean it that was forensics, right? You know for now we have a lot more things we can look at but still reversing and looking for fingerprints and and stuff is is a huge piece of the of the forensics you know puzzle but you know software supply chain security that's bombs.
You can't, you know in Security today. You can't work 10 feet and not hear that term. And it certainly one of the biggest things are the other hand.
Unfortunately, you know, we've had some very high profile incidents. around software supply chain, where You know. Malware players were able to actually insert.
Malware and into software as it was being compiled. Sometimes using the very tools Right that that we used to to do cicd. And and of course the results are pretty Kind of strophic in well, I don't think catastrophic but the results are widespread and they're very very serious.
From where you sit and you look you have that reversing engineering background Thomas love where do you see the like the issues and the problems and feel free to you know, use examples now sure I mean selfless supply chain security is a really difficult space because a lot goes into building the software right? Like you have the source code and the source code can have weaknesses and those weaknesses can become vulnerabilities. Then those vulnerabilities could be misused and you know become, you know, exploited then you have you know, malicious actors injecting real code into the builds.
For example solarwinds case and some verse was exploitation of a build system to inject a malicious code during the build. Then you have people abusing soft distribution sites. So if you think about Cold Case, which is a more recent one that is leaking secrets and then misusing those secrets to gain access and then plant your malicious scripts for Example then more recently a couple of months back have Circle CI which is somebody attacking a developer an actual person developing software to gain credentials access all of the production systems the build systems take customer Secrets start abusing them and then potentially even replicating, you know, what happened with solo and just you know, using the secret access to build builds a systems to inject malicious code or whatever and and yeah, that's that's kind of the state of software supply chain security.
It really is difficult because there's so many entry points and that's why things like as ball matter because you want to know what's in the software and that can be open source that can be your own stuff that can be secret like all sorts of things, you know can't come together to make a piece of software and all those need to be secure. Absolutely, you know Thomas Sullivan my time and I'm a little older than you. Okay, even though we both been in security around the same time in my time.
I think the biggest difference I see. in the way, we do software and what kind of results in the software supply chain is incidents is You know, I call it the age of Frankenstein software and I don't mean it in a bad way. But today so much of the software that we published that we deploy.
It's not original code written by a developer or a small team in developers who then put it through some rigorous testing and so forth today so much of our software has is made up of components stitched together that have dependencies in those dependencies have dependencies, right? And so when you look at like s bombs and so forth Supply chains, I think one of the issues are is it's not just the software in the code. It's what that software in the code is calling outside of the software and what that's calling, right?
And so those dependencies, you know, and then you combine artifacts and components that maybe were never combined or you know, we're in tested working together and that could create yet new. Potential problems, right? and so You know when we talk about software supply chain Security today, it's not even just the software necessarily.
That we can see in the source code. It's it's software that it that gets called by that software as well. And to me that that makes it.
exponentially more complex to kind of really wrap your head around. Yeah, and I mean, I like to think about it in Stacks, right you have you know, your user interfaces. You have your backend Services, you have your databases you have dependencies.
We treat files compress things, you know, all sorts of things have to come together to create a more General application and then you either kind of deploy it as like a model tap or there's a bunch of microservices all talking to each other, right? That's the other. Yeah.
Yeah, and then you know, like if you just think about as bomb I think it's one aspect of the problem. That's the list of ingredients but if you have these services and they're talking to each other there's data flows and some of those data points are actually, you know, more critical than others like they and they can contain private information or you know contain, you know, valuable straight Secrets or things like that. So I think like as bomb is just the bottom layer that eventually evolves into you know, all now you need a data model of how These things interact because if you think about log for J, for example, if you have an exposed service and it's logging things, right you can have remote code execution which obviously is it is bad, but, you know knowing where that sets how does it talk to other systems and what could be exposed is actually the type of risk as ISO is looking at right not necessarily all we need to patch log for Shell you do but if you know have so many resources and you can prioritize patching only so many things at once which ones are most critical and you know, protecting data and protecting, you know actual users is the most critical bit.
So that's why I think as well as a little bit lacking because they'll tell you what it is, but they don't really tell you how important it is. agreed a great you know, you're talking in a million thoughts going through my mind with this but I I agree that's Farms are are a piece of it. I I think sometimes though.
We just lose sight of the fact that cicd and now we put in the microservices and these things and then and even the way you know, you look at Cloud native architecture a lot of new apps, right or going out in this Cloud native architecture. They all can trip you to making. It's it's not an easy thing to do.
It's just not easy. Right and Security's never been easy, but this is certainly not easy though, though. We're trying Tom Tomah Thomas Thomas reversing Labs.
How do you help with this? hmm Well, we like to think about application security a little bit differently. We do like to think about it as software supply chain security meaning analyzing applications and figuring out what what's the worst which could happen.
What are the embedded risks? Right and we kind of Define six categories of risk which are important to create and ship secure code and we like to say, okay. So, you know, you have to be compliant meaning that you're not exposing any secrets that anybody can just misuse and embed malware into your code or get user data.
You have to respect the licenses of whoever solve for you using to build your own stuff. So that's number two. You don't have you have to kind of patch vulnerability.
So you have to look at that. You also have to properly configure those soft words. So it's more resilient and then it doesn't allow a vulnerabilities to kind of explode into exploits.
You don't want to ship anymore. Definitely. It and you don't want to allow that software that your shift to be tampered with like so those six things are what we are most concerned about when it comes to software supply chain security.
And yeah, like if you look at all of those things together, you get a risk profile of an application and if you're developing it you kind of get a pointer. Oh you need to fix this first because if you don't patch this, you know bad things could happen if you're shipping malware bad things happen. So on so forth, but we want to be conscious of you know, the amount of time it takes to fix all these things and just catalog them.
So we want to prioritize those things and offer a mediation advice. So that's how we like to think about it really it's about identifying risk helping people mitigated and tracking if they're successful or not at eliminating risk. You get it.
All right. We don't have a lot of time but I want to make sure people kind of how do they how did they engage with you? Not only personally but with Regency left.
Yeah. Well, you can visit the website. Obviously.
com has a whole bunch of information, but we really are building something very unique on a page called secure dots offer and it really is about this mentality of you know, building a secure piece of code or using a secure piece of code and it's kind of like a teaser very soon. We will have an open source solution put out there which itemizes all the different aspects of risk for software supply chain Security in open source, right? So given any open source package, what can we tell about you?
What would be your risk profile if you were to take this and very excited about that it's coming so I don't want to get you in trouble and say more than you're supposed to but when it does come out, you'll come back on and tell us about it. Okay, you know application you mentioned application security and I I think back about how applications security has changed over the last 15. 20 years he mentioned something before about microservices and the flow of information maybe even Secrets, right?
And and so much of that is now governed or managed by apis. and you know, so this whole notion of API security has become such a it's such a it's really part of application security. It's a big part of application security now right for both internal and external apis.
I think we can call them that so much I think oh I just came out with like API. Vulnerabilities, you know API kind of top 10 list. What about that at reversing Labs?
Is that something you guys are looking at? Yeah, I mean very concerned about Secrets right because Secrets can be misused to you know, both inject malicious code into the releases to even misuse systems. And if anybody's doing really, you know correctly cicd meaning the deployment part is continuous if an attacker was to misuse a secret they can really push malicious code directly into the production.
So imagine somebody, you know, like hosting a large website and they're developers are constantly committed code and that code gets published to production as soon as it's you know past all the unit tests and all the checks and attacker if they have an access to a secret and that allows them to impersonate the developer they could do the same thing and they can access the whole user base which is like any but any malicious code they like so that's why Secrets matter and and I think that, you know hardcoded Keys credentials only other things they're all over the place like you can have them in your gith. Repo you can have them in the code. You can have them injected during the build.
So just properly cataloging them itemizing them and making sure they're properly rotated. All the time is very very difficult. So one of the things we built lately is just to look at the application as it is and then decompose it and figure out all of the secrets within the application package and people get very surprised at how many Secrets they're actually shipping right and with it a bit of a research to figure out are all those Secrets important, right and turns out like 95% of all the secrets which you end up shipping are not even your own.
It's all just coming from open source and all the other things as we were talking before right you pick up all the open source libraries to build your own app. Well that comes with secrets and now you're asking your security team to three hours all those secrets and figure out which ones matter and night if I percent of them don't because they're just testing he isn't certificates. So like in traditional security not supply chain you would you you all large fatigue, right?
Because you get all sorts of blinking red lights and you have to figure out what that's what it is on a software supply chain security as well and having the right tools is difficult, right? Got it. com.
Is that the right URL? It is. Yes, you'll find everything there.
Probably five. Thank you for being our guest here on techstar TV today. We appreciate it.
Hey, when this open source thing comes out, you'll come back on and tell us about it, right? Absolutely. Talk to you soon.
All right, I'm gonna hold you to that. We're gonna take a break here on Tech strong. We'll be right back with our next guest.