Bridging Open Source Analytics and Community Engagement with Scarf’s Avi Press at OSS Seattle 2024

This is techron tv. Hey everyone, welcome back to our time here at Open Source Summit in Seattle. Lots of great conversations 'cause we have so many people that are involved in contributing and even built careers around open source and being part of projects and maybe turning that into businesses and that kind of thing.

So actually our next, uh, gentleman who's joining us today is AVI Press. Welcome, Avi from scarf. Thanks for having me.

What, what, what do you do at Scarf? I'm the founder and CEO Founder and CEO. Okay.

Good thing I didn't say something else. So now it's all good, good to be talking with you. So tell us about Scarf before we kind of get into some announcements and things happening.

Yeah. Um, so Scarf is uh, a platform for usage analytics for open source. So if you are an open source maintainer or maybe you're a company that, um, builds open source, um, scarf is a way for you to understand how that software is being used.

Mm. Where in the world is it being used, how many people use it, which companies use it. Um, so it's a pretty comprehensive platform, um, to address those kinds of questions that lots of projects have.

Um, and we, uh, as of recently are a partner with the Linux Foundation to kind of provide these services across, uh, all projects in the foundation and just help have more data-driven, open source development and, uh, maybe less burnt out maintainers too. Hey, that's always good. Yeah.

And I haven't been part of an open source project, but I gotta imagine you see downloads or all the downloads that are coming and people that are involved on whatever channels that have communications that you've got, but you still really don't have a good handbook for who all is using this and is it, you know, are there things we could be doing to help those people even more than we are? Is that the kind of data that you're seeking out? Absolutely.

And like yeah. 'cause you know, the people that actually come talk to you and say, Hey, I'm using this and I'm having a problem, or Hey, I'm using this and thank you, is very, very small portion of the people that are actually using the software. Mm-Hmm.

And so, um, you know, you have both people that are maintaining projects thinking that it's not providing value for the world. In fact, it's providing a ton of value to the world. Or maybe you get a spike of a hundred thousand downloads that all came from one machine downloading over and over and over and like you get a lot of misleading signals or you get signals that are a very small part of the whole picture.

Yeah. Um, and so I think like, yeah, we exist to try to help fill that, that gap and try to give a more, you know, comprehensive view of what's actually happening with the software. Do do any examples come to mind of situations?

You know, projects that once they had that data shifted or changed in thinking? I mean we've, we've worked with, we've worked with, um, businesses that have been sold on the back of the data of like, oh look, a lot of companies use this and then they can go it acquire. Oh, interesting.

Really validate that. They're absolutely this is, there's a market for the Yeah. And like, you know, fundraising rounds that happen because like, well we don't have a paid product yet, but look, you know, 10% of the Fortune 500 has been using this or, you know, what, whatever, uh, what, whatever those things might be.

Um, and I, I think that one thing that I really, that I find really rewarding also just like outside of the, you know, like business sphere as well, is just being able to show maintainers that like your work matters and it has an impact on the world difference. Differe then like, 'cause I think it's, it can be a very thankless job. The, the only impetus sometimes people actually reach out to is if they have an issue and they need to, they get it fixed and they're not happy about it.

And, um, I think kind of more proactively showing the other side of out of like, Hey, this stuff's actually very, You know, my view is I think even leaders on even internal software development projects grossly under appreciate how important that is to people. 'cause you're, you kind of feel like you're in this part of the submarine underwater at this depth working on your part of it and you're kinda go, does this really matter? Am I doing the right thing?

Am I working on the right stuff? Yeah. And just having some kind of validation of that to say, in addition to the person walking up to you at the conference saying, you know, Hey, love your work.

You know, thank you that, that's super meaningful. But having some data too to validate that, Right? Yeah.

It also helps you be a little bit more proactive as well. I think that that's one thing that I found very exhausting about being a maintainer is that you just never, you're always like catching up to, you are responding after the fact and having to be very reactive. That can be really draining.

Absolutely. Um, as a maintainer. And I think that, you know, if we can prioritize our time more effectively, focus on the right, prioritize better, these kinds of things, it just can help make maintenance a little bit more tractable.

How does that data get communicated to people on an open source project? Do a few people see it or is it kind of something that's shared widely or So on Scarfs platform, the way it works is that, um, you know, by, by coming to us with your project and wanting to track it, that data is yours to, to distribute as you see fit. Um, so by default you are the only one with access to that data.

Um, a lot of projects tend to share the, the metrics with their communities and um, yeah, we see lots of examples of like, you know, drive your community meetings with data about what's been happening. Are we growing, are we Shrink? Nice way to say, Hey, we you got, we hit this number.

Yeah, exactly. Or like, Hey, like a lot of people are using an old version and they haven't been upgrading very well. What's going on?

How can we help them? Like there's a lot of, um, yeah, there's just a lot of stuff happening in the community that you're not really gonna see because most people are not gonna come talk to you. Yeah.

Yeah. So I know you had an announcement recently. Do you wanna talk about that a Little bit?

Yeah. Um, so yeah, one thing that we just announced is that we've partnered, uh, with a company called Common Room. Um, common Room provides kind of a, a one place to aggregate lots of different, um, go to market signals for businesses.

So, you know, tracking who's talking to us in Slack and who's starring us on GitHub and opening issues on GitHub and these kinds Of, so kind of metadata metrics about what's going on in Yeah. Community. Exactly.

Um, like a lot of go-to market metrics, very community oriented. Um, and one thing that um, I think a lot of common Room customers had been missing is, well, what about the people actually using the software? Can we get that?

Similarly, scarf does not do a lot of these other community things. And so it's very complimentary in these ways and, um, we're very excited to be partnering with the Common Room folks to be, I think when, when Common Room is paired with scarf, our mutual customers can get a much better picture of the whole user funnel. Mm-Hmm.

The time people discover us to the time they become a paying customer. Like what happens in between. And we can help provide a fuller picture of That.

So this gives scarfs users and customers access to the broader common room. So specifically what this means is that, um, scarf customers can seamlessly export their scarf data into common room. I see.

Okay. So, you know, if someone starts to get involved in forums and they're like, is this per, like what version of the software is this person using? They didn't say, um, we might be able to, you know, actually plug those numbers in and um, kind of give, give a fuller picture of both how they're engaging, but also how they're using the software itself.

Mm-Hmm. Or how their team is using the software itself, which maybe that person doesn't even know. Um, and so yeah, I'm really excited about it because we, you know, we really focus on the one thing that we do very well Mm-Hmm.

But, you know, how do we help our customers? Kind of that's what part It really makes sense. Yeah, exactly.

Kind of do, instead of doing a little thing for everybody, try to do a few things really, really well for people. Exactly. That is what we have focused on.

But um, you know, we're just one piece of a broader puzzle. And so kind of building those partnerships with other folks in the open source spaces for, um, very valuable. So, So gathering all that data are, are there some insights that you would say if, if I could advise any new or or existing open source projects?

There are a handful of things that if you did, would make a huge difference or make, make a measurable impact. What would those things be? A few of them.

Yeah. Um, oh, so, okay. Great question.

One I should say is I'm giving a talk on this exact topic tomorrow, so you should definitely come it out. Yep. We check it out.

It's being recorded. You can check it out. Yeah.

Um, so a few things. I mean, one I would say is making sure, I mean obviously like making sure that you are tracking these kinds of things so that you can actually measure the effectiveness of the things that you're doing. Yep.

Um, you know, I think that's kind of an obvious one to, is that Like user metrics or what, what would be an example of that? Yeah. Usage metric.

So like for instance, like what is the, uh, given that someone landed on a readme, what's the likelihood they go on to download the software given that they, on this page, what's the likelihood or, you know, who's having a hard time upgrading from this release to the next one? Maybe it's like Windows is, maybe it's, you know, some other specific combination of factors that will be very obvious if you're actually just tracking and looking for this kind of thing. Um, I'm trying to think.

I mean, one thing that has been um, you know, coming up a lot is that the numbers that people, their download metrics on registries, if they just see, oh look we got a hundred thousand downloads this week, our data would show you probably actually have about a 10th of that as actual users. Mm-Hmm. The rest are bots and things or What?

Or just people downloading over Cycling Or Yeah. Like yeah, between, between bots and repeat downloads and these kinds of things. Like you probably have much less of that as actual users.

Users. Um, we see that, um, people are not gonna upgrade very often as much as you might like to think. Yeah.

So you may put out that security patch doesn't mean people are using it. Right. Um, and I think that, um, yeah, these kinds of trends about like version adoption and how we can better incentivize and reduce friction towards upgrading is something that I think is very important.

Both for maintainers, also security practitioners, these kinds of things. And just people at businesses that rely on all this stuff. And so, um, yeah, there's been kind of a lot of learnings and a lot of surprises of this data.

Um, and so that's what's been fun to kind of share what we've learned from analyzing lots of downloads stuff. Yeah. Yeah.

It's, it's, um, it it, the, the, if you build it, they will come as kind of an exciting, passionate model, but there's also a lot of value in saying, um, well I'd love to know if they installed it and actually are running it right after they did. Or maybe it took them three downloads before they got it right. Or they tried three other times at this version.

Well why did that work? You know, give you some insight to what people are doing with your software. Right.

Yeah. What they're actually doing with it being the really like, um, yeah. Tough part to glean.

'cause I think we, the open source community broadly, I think operates through a lot of proxy metrics that are not really the thing that you care about. Mm-Hmm. And so like, we have a thousand stars.

What does that mean as, uh, what do we do with that? Um, you know, I think we're, the thing that we actually care about is like, does this have an impact on the world? Like, can I build a business around this or like, should we keep working on this?

Or whatever that might be. Um, and I think the, the closer we get to what we actually care about, the better off everyone is gonna be and everyone's gonna get better software as it result. So Yeah.

That has, how soon did the upgrade is an important metric 'cause it's, we all know we've used software that great new things coming, but man, to get from where you are toward that new version, you know, we all have our own technical debt or just even resources at times. Yeah. You get there can be really daunting.

A lot of people are never gonna upgrade. Mm-Hmm. That's the kinda the reality of that too.

Most people wouldn't upgrade these things if it wasn't automatic. Right. Exactly.

Exactly. Um, and so like even if you reduce a lot of that friction, just some people they have the software they have and they're not gonna download it again, then that's kind of all it is. And we need to just be more accepting of that or more, you know, just acknowledging that is the pattern of behavior and that's kind of the way it is.

But, um, So you said you're doing more than one talk, is that right? Yes. What else are you talking about?

Um, yeah, so tomorrow it's about like what lessons learned from like the last billion downloads on scarf. Um, the, I'm giving a talk on Thursday about what, what information, um, package and container registries have that are hiding from maintainers. Like they see a lot of stuff.

They don't expose a lot of stuff. So like we'll dive into like what Sounds like a 60 minutes episode or something here, you know? Yeah.

I mean what the package containers I know that you don't know. I mean, they Know a lot. That's the thing.

Like they see a lot of information. They see, you know, they see a lot of IP addresses, they see a lot of header information, like a lot of rich information there. And you know, our view on this is that if maintainers had more access to that kind of data, we would have more effective maintainers and we'd have better open source for everybody.

Do you think it's 'cause they don't make the effort to share it or they don't want to? 'cause there's value in them knowing that, but others not? I Think it's a bit of both.

I mean, I think that there's a lot of registries. They just, um, you know, processing and analyzing this data is a lot of work. Mm-Hmm.

Um, it's, it's, it's a lot of work, a Lot of Resources. Right. If there's no business incentive for them to do it, it's not gonna happen.

Right. Um, I think also like attitudes towards analytics, the open source world also have been a historical hindrance for this kind of thing. But I think, you know, the work that Scarf is doing I think has been shifting attitudes around this about like, hey, uh, analytics can be collected in an actually very responsible manner.

We're GDR compliant. Like there's a lot that you can do to, to collect this data responsibly, um, you know, get the right opt-ins that you need to do the whole thing in a good way. Um, and to ex and uh, expose that to maintainers.

And so I think that it's a shame that more registries are not kind of, their businesses are not set up to incentivize that kind of work to be done. Um, and so that's why we exist. That is our business model is that's The niche.

That's the need you're filling. Yeah. Yeah.

And so, you know, we work with a lot of companies that commercialize open source or universities that build lots of open source or these kinds of things where you have a little bit more of a commercial interest in that data. And so we're happy to, you know, provide that where we can and still like work with lots of, you know, completely non-profit organizations and help them do a better job as well. Well, good luck with your talks.

Thank you so Much. They'll be available online. Yeah.

I think you made a smart career choice. If you've gone into media, then your name badge would say Abby Press Press. Yeah, that would, that would be confusing.

Disaster. Yeah. So and they really wouldn't talk to you then when you go into their booths, so.

Right. Fortunately I just get all the benefits of Avi Press on my badge. They can get into a lot of nice parties that a lot of free food.

Pretty smart. Yeah. Good way to do it.

Hobby Press, CEO and founder with Scarf. Thanks for joining us today. Thanks For having me.

This was fun. You bet. We'll be back soon.