PyPi Token Leak with Shachar Menashe

This is Textron tv. Hey everyone. Welcome here to Textron tv.

Welcome back. Our next guest is, uh, from our friends at Jay Frog. He's one of the frogs I want to introduce you to Chaka Min Minh.

Neha. Hi Nick. Could you believe I messed it up?

Shahar Minha, senior Director, security Research at Jfr. Shahar. I apologize.

Say your name right for us. Uh, Yeah, it's a tough one in English. Uh, yeah, it's Shahar, Minha Neha.

Hi guy. It's not that hard, really. It's the, you know, a lot of people in Yeah.

You need, don't, they don't have the cup. But, uh, you know, I, I've had a little practice. Anyway, Shaha, your senior Director of Security Research at Jake ffr.

We're gonna get into the whole Jake Fart security research kind of team and, and, uh, mission. But before we do, let's hear a little bit about your background. Uh, sure.

So, uh, I'm in the secu, uh, I love security. I'm in this, uh, security space for, uh, I would say, uh, roughly 20 years already. So, quite a lot.

You're in here. Uh, I've been doing, you know, binary reverse engineering, hardware hacking, like pen testing, uh, writing, automated tools for security, like really everything. Uh, I got a lot of time to work on it.

Um, I joined the Jfr about three years ago from the VI acquisition, uh, right. If you know about it. Yeah.

So, uh, yes. Well, I know about it, but our audience may not give us a little background on that. Yeah.

So, so about three years ago, uh, JFR decided, you know, they were primarily, uh, the Rob's company and not yet DevSecOps, and they wanted to expand and become a DevSecOps company, so really focused on security. So, uh, there was a startup called voo, which focused on, uh, binary analysis of, for security purposes. So you take some sort of binary file, like a formal image or a Docker container, or just something compiled and you scan it and then, you know, uh, to make security conclusions from it.

Like if, if the binary has a zero day vulnerabilities or susceptible to cvs. Uh, so this is what, uh, the previous, so this is what we do did. Then I worked there about three years ago.

Uh, JFR decided they want to get more in the security space, so they, uh, acquired, uh, the startup. So, uh, we were about a hundred people. And then we joined Jfr as, uh, the security team and helped, uh, basically build jfr, uh, security scanning tool, which is, uh, X-Ray.

Sure. Uh, and now, you know, yeah. Merged our, uh, previous technologies there.

And now, uh, well, we are here for a while now. So we're really a part of, uh, J Rog. Well, J Frog's also made some other acquisitions in the security space, and, and they really have become, as many of the DevOps companies have the become DevSecOps companies.

Right. 'cause you, at some level, DevOps is DevSecOps today. Um, now look, it's one thing to do, you know, security.

It's another thing, of course, to do security research. So JFR has maintained, and it's now called the jfr Security Research Team. If you would've, my, actually, before we even get there, Shaka, I, I think we, our audience knows Jfr very well, and this evolution from DevOps and DevSecOps, we've also covered a lot.

So let's jump right to the Jfr security research team. Kinda what's the mission there? What's your charter, you know, what, what, what are the goals of that research team?

Yeah. Uh, it's a good question. Uh, we have multiple goals.

So, so, uh, I guess the first goal is to enrich our product with, uh, insights that, you know, you can't get automatically. So, for example, one thing that we do, we have three teams. We have, uh, a CVE research team.

We have a zero day research team, and we have, uh, malware and secrets, uh, research team. So these are all separate teams, and I'm, I'm in charge of all these teams. Um, and all of these teams both do, uh, you know, feature enrichment in actually doji for product, but also do some, uh, community work, which actually the, the Python, our token relates more to the community work and also just pr, so public relations, you know, just, just to, uh, to make sure JFR was known that now we're Dev TechOps and not just DevOps.

Uh, and we do a lot of effort there. So, for example, the enrichment is that for, uh, you know, each CDE that the products finds, we don't just supply the public information about the cv, but we also have our own insights, like when exactly it's exploitable and like how to remediate it. Uh, actually yesterday we published a blog on a very new CV that came out, and we published a new way to exploit the vulnerability, so people will know to watch out from that.

So, so these kinds of analysis, uh, also things that we do responsible disclosure. Uh, so it's really a multitude of things, I would say. But, uh, a mix of like helping the product with enriched, uh, information and pr I would say.

Very cool. Very cool. All right.

Let's, um, let's jump into this instant, uh, disclosure here that you guys came out with, uh, about a token, an access token that, Yeah. Uh, Tell us about it. Yeah, so I think actually the, the background story is almost as interesting as the finding itself.

So, uh, really two years ago, we built a secret detection engine. And the idea is that, you know, you upload some docker container or, uh, whatever package you want to, uh, or a factory, and then we check that you didn't forget what are your access tokens there, that if you have look, uh, secret there, then we alert about it, and then you can't publish it. So the feature was built, but we, as the research team, we were tasked with understanding how to implement this feature without, uh, having false positives.

So what we said is, oh, okay, we'll, uh, build a very big database of, uh, real world data of packages and containers, and then we'll run all the secret detection on all of them, and then, uh, we'll see if mgen got false positives or true positive or whatnot. So this was, all of this was that dog fooding effort, which is pretty funny. So, so we did that, and we ran that, and we saw some, you know, we saw resilience, we saw good results, bad results, and the bad results.

We fixed the engine. But, but what we didn't expect was the number of, uh, true positives that we found. So, so the database that we built is basically all of the darker containers in Dock.

We, all of the packages in NPM Pipi, like a huge database, it's 8 million artifacts. Okay. And it's getting up into daily, by the way, still.

And what we saw is that we we're finding, uh, a arrival hits. So we, we have like 200,000 result, and a lot of the results we can actually pinpoint and say, oh, wow, this access token belongs to some company. Like, I'm, I'm just gonna throw it out.

It's not a real example, but like, Microsoft. So we, we didn't just find access tokens, uh, because of this, uh, false positive test. We, we actually understood who they belong to, and then we thought, well, this would be a great community effort, uh, to disclose these.

So basically from two years ago, uh, each week we do at least one disclosure, uh, when there's a, yeah, it's, it's crazy. And the tokens, uh, the, like, the parallel token was the most crazy one that we saw in terms of, uh, access, but not by far. We saw tokens of, you know, fortune 500 companies and all sorts of, uh, maintainers and companies that, you know, tokens that with full access to the code base or leak a lot Krill, VPII, uh, it's really a serious problem.

So basically each time we understand who the token belongs to, we do a disclosure to them. Uh, I would say actually that the biggest problem is getting in touch with them because it looks like a spam, you know, without a token of, again, Just, well, I know, I, I, I will tell you here at text for instance, we get people who, you know, give us the, these not so much sleep. I don't want to get into too much of our infrastructure 'cause people will use it against us.

But that's, we get, and, and, and the thing is, most of our infrastructure, we're sa we're using SaaS tools, and it's not even our infrastructure. Yeah. So you're gonna tell me about a WordPress vulnerability or WordPress plugin or something.

There's always, I, I could shut the, the plugin down, but I'm not Yeah. Plugin. You're not actually running it.

Right. Um, but we get a ton of emails and, and separating the spam from the phishing from the real, real reports is not easy. But you know, to me, so look, I'm in security almost 30 years, right?

I wasn't always on media. I did venture backed companies, one of which was a software company called Still Secure in Boulder, Colorado. And we were intrusion prevention, vulnerability management, network access control, right?

Back then back, it's a big thing. Yeah. Um, and one of the, one of the like cardinal rules, like our product, they have vulnerability access.

And this is before we would scan and we only scan production. We didn't scan pre-production. Right.

You were lucky if they scanned at all, but if you did scan, it was, you know, infrastructure that was deployed already and apps that were deployed. But one of the biggest problems we, we, we had is the, the security team, they would look at it as job security because they would find so many vulnerabilities like a book, and then they would have to do remediation that by the time they were done, by the time they were done, ready for the next scan, and they would only scan once a quarter, twice a year or something like that, they, they didn't even finish the remediation from the last scan. And so it was almost like job security scanning for leaked access tokens that were left behind, or whatever it is, in a way, almost like job security, right?

Because you're finding so many, well, what a week you're, you're going, how do we, how do we eliminate this altogether? Yeah. I mean obviously leaked access tokens are a problem.

How do we take that outta the equation? Yeah. So I would say actually, so for us that rescan the whole internet and, you know, um, disclose to everybody, we have a lot of work to do.

Like for us, it's, it's a lot of work. But for, you know, a specific company that scans for access tokens, uh, that's actually the, it's not a big problem. Because for them, I would reckon they would find an access token, uh, maybe once a week or once, once a month or something like that.

Really? So, so that's, yeah. So, so I don't think for, for an individual company that scans their own things, I don't think it's a problem.

Uh, I think that CCBs are definitely a problem in that regard. Well, there's so many, I forgot what the latest number I saw how many CBEs are being Yeah. Added every week.

I think it was, was it eight 900? Some crazy number. Yeah.

I know for us each day, like each day there's about 30 CVS added in all relevant technologies. So I wouldn't be surprised if it's, No, I think a number was like 800 something. It, it's crazy because we find more CBS than, you know, than ever.

It's not like we're finding less cbs. Yeah. You said that we're finding less vulnerabilities that get a CBE number.

That's true. Right? Yeah, there's more each time, but I think in secrets, it's actually, when you are scanning your own things, it's actually not a problem.

Uh, so the, the, the solution is just to actually scan it and have something, you know, before, Well, maybe it's as simple as that, right? Before you guys scan it, they should be, you know, people should be scanning that. That's good ig, right?

It's good cyber ig. Yeah. I think, uh, in Bruce Case, it was interesting, uh, coming back to the Python League token, uh, because, uh, if you read the, the main Commander's blog, uh, he did actually, uh, employ, uh, a scanner and a scrubber, but it only worked on source code.

And the thing here is that the secret was leaked, uh, like they added it in source code, the date of globe, and then we removed it from the source code. But then of course, Wait, we still add there? Yeah.

Yeah. It remained in the build. Uh, and because again, we, we come from the binary world, uh, like that's our, and because of that, we scanned for secrets in the binary as well.

And that's, we start only in the binary. So, so that was super interesting, and I think that's the case why, you know, it was left for a year, about a year and a half there. Right.

And, and I think that's the case why it was left for so long and nobody ever found it. Yeah. I mean, so I mean, moral of the story is scanning your secrets is good.

You should be scanning your binary secrets as well, or scanning secrets. No, for sure. You know?

Yeah. But I think another moral is, uh, there was a few mistakes that were done here. For example, uh, if you're familiar with like the Google South South framework, uh, there's a requirement there that, you know, the actual, uh, build happens, uh, not under the variable machine, but, uh, on a, on a clean, uh, third party machine.

Like, you know, like you have Jenkins, uh, deployed to some, uh, clean machine, and it, it, uh, clones the source code from there, and does it build from there? And, and if that would've happened, then, uh, this token wouldn't have been leaked as well, because the, the other problems here were that first of all, uh, the token was inserted into a source code, uh, hard coded, like even for a second, it shouldn't happen because then, you know, it gets saved into all sorts of places. Like if it was left as the environment variable, then it would've been okay.

So both the fact that it was inserted, hard coded, and also the fact that the build happened on the development machine and not a clean third, you know, clean, uh, external machine. So it's all of these things together. Agreed.

Agreed. It's crazy. Well, look, first of all, thank God you found this when you did.

I mean, it was, it's a year and a half later. Later, but still a lot more, a lot. You know, we, we save a lot of aggravation and trouble with these things, and, and it reminds us, as I say, it's, we thought it'd be vigilant, right?

Security research teams, not just the Jfr team, but many of the security research teams in the cyber industry do great work day in, day out. A lot of it we never hear about, you know, at the, at the PR level here, like this, right? At the public level, a lot of it is behind the scenes stuff.

You know, that that's one of the issues in security. When you're doing a good job, no one knows because nothing happens. Right.

And, um, so keep up the great workshop. That's, it's good stuff. You mentioned you're gonna be at Swamp Up this year?

Uh, no, not for sure. But someone from Security Research will, uh, sure. Be Swamp Up is in Austin.

I, I wanna say it's Eptember 10th and 11th. Yeah, Yeah. In Austin.

It's, uh, from uh, nine from September 9th, Right? It starts the ninth. Yeah.

Which is I think a Monday. Yeah. And goes through Wednesday, something like that.

Um, I will be there. We'll be covering it. I'm looking forward to hearing all the latest and greatest from my friends at the Frogs.

Say hello to everyone. Thank you for coming on Techron TV today and keep us posted going forward. Awesome.

It was my pleasure. And uh, we will. All righty.

Shaha, senior Director Security Research at Jake Frog here on Techron tv. We're gonna take a break. We'll be back.