Sal Kimmich, Sonatype | Open Source Summit Europe 2022

This is texturing TV. Welcome back folks. We're at the open source Summit in Dublin, and we saved the best for last we have Sal kimmich who's head of developer relations for sonotype, and we're going to be talking about open source gender equality and how it's really just all a math problems.

So walk us through how all this comes together in your mind, but there is a way to solve this issue I think so absolutely. So I've been an open source for about 10 years, but before that, I think I had a really different background than a lot of people do I started out my career working on actually Boeing aircraft. And when I was looking at that problem, we were trying to look at the interface of the cockpit what information was available and how it was altering the ways that they were making their decisions in real time.

When I moved away from that, I started working on some really interesting Neuroscience problems. I really became fascinated about how quantitatively can we Define. What is the difference between a high and a low performer indecision making for different areas and contexts?

Coming out of that space when I looked at open source, I found it to be really interesting because this is a space where we've got to do really highly distributed decision-making around intellectual property. And we do all of that online over digital interfaces and I think that it's these digital interfaces that are really having a major impact on who is allowed to participate when I first became interested in this problem. This is this is pretty new to me.

It was just the last open source Summit. I went to the diversity track and I was hearing a lot of the same old discussion about how we need better hiring practices. We need more mentorship and all that's absolutely true.

But there is a study done in 2017 that looked at real GitHub pull requests from male and female identified handles. And when you take those genuinely real pull requests real code and you split them up and you say let's review these without any identifying information no images no handles no email. When you have none of that and you just look at the quality of the code review you saw something fascinating.

So the statistics look like this without identifying information female identifying GitHub handles actually had an acceptance rate that was 4% higher than the average male PR acceptance meaning that without any identifying information. They were providing higher quality contributions that were statistically more mergeable. With those exact same pull requests in this study.

If you did provide identifying information about the gender of the individual those exact same pull requests from the exact same people with the exact same code. The female PRS were then rejected at a rate which was 9% lower. So we're talking a 15% Gap in the acceptance rate of female contributions when we're looking at literally the exact same code.

So those statistics alone prove to me two points? There is absolutely a bias that is going into the decision-making around what code is accepted that is not germane to the code itself. And it also proved to me that if you remove that metadata, we solve the problem and for me, I think that's a really interesting approach because yes, we need much better diversity in this space.

But when we're looking at unconscious bias from human beings, it's not something that you can teach out of anyone you can teach best practices. You can talk about diversity all day. But we could also just simplify the approach and see if we can engineer a solution that makes it easier for maintainers to make the right decision.

And I think that's a really really good direction to take this. So, I mean first of all, I'm not surprised because coming from the male side the world you're basically telling me that 15 to 20% of the people on my side of the equation or idiots. So I get that I'm okay with that.

The next thing though is so do we just strip out the identifiers in our workflows so that we don't have this issue and is it really that simple? I think it really is that simple. So one thing that I want to be really clear about is that I would not argue for anonymizing.

Any other part of Open Source? I am specifically asking that we mask the metadata at the initial point when I pull request is received egg after you push a button as to whether or not you want to accept or deny that pull request all of the metadata should become available immediately. I do not think that open source can or should be the strange Anonymous space.

That's not how open source works or how it should work. But we need to make it much easier for people to make decisions based on the quality of code and not based on any other assumptions that don't have any relevance to whether or not it's going to improve the code base. Outside of the example at hand codebase.

Do you think that this same kind of dynamic Works in other Industries and other places? Yes. Thank you for asking because it absolutely does one of the reasons why I'm so confident that this would work is because I actually did solve this problem in a different field a couple of years ago.

I was working in computational neuroscience and I do miss my supercomputers, but for a couple of months, I did not have access to a supercomputer and I wanted to look into cloud computing. There was no resources for that at the time. So I designed a large online course that had about a thousand people that took it the first year and we had experts from all around the world come in provide educational resources to teach people all the way from the beginning of like what is a Jupiter notebook to how do you do generative Intelligence on stage maker?

It was really really cool course called the online brain intensive but with my signal processing background I decided that on all Of our communicational platforms slack Etc. Even the experts themselves would not identify themselves. So it made sure that the conversations that were had were much more interesting because nobody knew when they were talking to the expert so there were much more respectful.

But much more interesting than that. This is a field with 17% female representation and computational Neuroscience. But my course had 56 female representation.

So the majority were female identified but much much more interesting than that, which I did not expect was that when I went and asked the experts each week. Who had the most interesting unanswered hypothesis? I was able to trace those handles.

back to women more than 80% of those responses. The experts said more than eight out of 10 times that it was a woman who had the most interesting unanswered hypothesis. That really floored me.

I initially thought that I had just inverted my stats right because 20% female representation 80% Females with the best ideas. How could that be? I thought that I had created a training platform that was primarily targeted towards PhD students if you completed it you got AWS credits to go compute and create a peer-reviewed computational paper.

That resource it turns out 35% of the women taking my course. Were 10 to 15 years ahead of me in my career. They were postdocs in their first or second postdoc on average now, why were women that are pretty much solidly in the middle of their career using my resource?

They cited it was because they needed access to computation that they were not receiving that their male colleagues were receiving. So this is a space where we're seeing implicit bias really interestingly. Make sure that we're knocking out some of our most qualified most interesting possible contributions.

And when we remove any of those factors, we make it possible for someone to go and speak to an expert just based on the quality of the information on the quality of their ideas. It allows those to surface. I don't know.

I would be tempted to almost reverse the biases a little bit and I'd want to favor the pull requests from women because you know be frank about it. You know, my wife is a lot more precise than I'll ever be and it's a Precision game at the end of the day. So maybe it's also just a mindset too that you know, if you're the manager of this thing and you're responsible for it.

Maybe you want to think through well who is putting together the most precise code and have a look so does this become a new kpi that we need to think through and really start tracking it becomes a little more interesting because you know, the people running this at the top of the food chain. They just want the best thing at the end of the day. Yeah.

Well, I would not argue for biasing towards any specific population. I want to build a platform that biases the best idea and I don't care what the body is that that idea comes in if we really want to get serious about globalizing open source and making sure that this is sustainable. We have to be serious about engineering Equity so we should not be focusing on any specific demographic and I think that's Really where I take issue with some of the current approaches to diversity.

They're really falling short of making sure that we systematically engineer a way that is sustainable and getting those best ideas. I do want to say the reason why that GitHub study most likely had better contributions more mergeable contributions from female PRS. Is because women tend to get involved in open source at a much later stage in their career.

They tend to be doing this from production experience. And so that's most likely why they're providing on average better commits in that space. How much of this does trace back to the way we approach math and grammar school?

It seems like girls seem to bail out and sixth seventh and 8th grade a little bit and it seems to be that the way that the math class is structured and the people who teach it come from a system. And so how much this is really just something we've ingrained not just in software development, but we've ingrained it all the way down into basic math stem and it just plays out across all these industries in a highly predictable way when you think about it exactly. Yeah.

So the pipeline problem is a really serious one and I do think it's that there is an issue with and we have to consider a lot of people think about these implicit biases is going One Direction right as a negative bias from one person inflicted upon another and limiting the opportunities of a class of individuals who are already underrepresented. That's absolutely true. But that bias Those both ways women sometimes can be biased against participating in spaces that they fear.

They will not be welcome in that's what is often cited is a lack of awareness of Open Source opportunities or a lack of mentorship to initially get started in open source, or a fear of Engagement from some of the stories and experiences that they've heard which are absolutely real. We're still looking at a space for the last time I checked the representation for women in open source is six percent. So when we're looking at an open source, it's significantly worse than what we're seeing on average and other stem spaces.

So there has to be something particular to the design of the way that we communicate in open source that is making this particularly bad. And when half the planet is made up of women, it's kind of like, you know, it's a little silly. We're missing the 54% of the best ideas in humanity.

Why are we doing that? If we all clam to want the best thing let's get that. Does this issue also play out for Nationalities and for race because if you know that and you know, it's easy to determine at least nationality or some sort of guess at that and does that play out as a bias as well.

I don't know if anybody's done the math on that but it seems like you know, I know that there are people from certain countries that just don't like or trust people from other countries. And so maybe the whole thing is playing out in other ways exactly. So unfortunately right now the only real verified study that we have is this one only looking at gender diversity and only looking at pool requests.

What I'm really arguing for am I call to action here is to get some major projects to participate in this metadata masking as a best practice. And if we do that what I would like to be able to test if is if pre and post we see an improvement in a couple of things. Yes gender diversity geographical diversity racial and ethnic diversity and I think you would also see an uptake in first time contributors that are likely to stay around and be retained because they see that they're in a space where they are being valued for their ideas and not their identity.

And you also think there would be less resistance to some of the programs. We're putting in place because we could mathematically support the improvements and the outcome and the output because right now a lot of what we're doing is theoretical and speculative and so people can be naysayers if they want because though, you know, they're basically proven to negative, right? Yeah.

So is that going to be part of the thing where we could actually have some math that says and here's how our efforts have benefited the thing and it's approvable Dynamic. Yeah, it has to be We cannot continue doing diversity improvement with qualitative metrics. We need quantitative metrics and if in my argument here, we find that it does not improve some of these demographics if we find that it does not improve the rate of contributor sort of community increase then I think that's great.

That means that that was a false hypothesis. I suspect from the study that we should at least be able to predict. We're going to see an improvement in gender, but there is no reason why we should not see an improvement for all other factors.

So how do people join into your what I've amounts to is a grand experiment. So, how will how do we get everybody kind of plugged in and working down this path? So when I really started looking into this, I mean my background is not really in diversity.

I just like building things that are good for Signal processing. So I initially put out this talk to OSS just as a 10 minute talk and then they asked me to do it for 40 minutes and I was like I will need more data. So I reached out to the new stack and they're doing a series of Articles reaching out and doing interviews with people who have already engaged in this space.

So there are several extensions already available one that was produced from Modzilla and one that is produced for Chrome. They're not actively maintained at this time, but they can and should be those are readily available for you to use as a maintainer on a project to put this into place today. Um secondarily, I think we really need to stand up and ask these platforms.

I focus on GitHub only because 85% of Open Source runs on GitHub right now, but all of these platforms should really be thinking about the cognitive design of the of the way that they're providing this information. In my dream of dreams. We have a button that allows an open source maintainer to choose to work in this way as a best practice and that's really what I'm trying to fight for.

It is possible that we may have to take some drastic approaches. org petition. We might have to do that.

But I think that this is so urgent and so necessary and allows us to get data on what does and does not work to get those best decision makers into open source. I want to see it. There are a lot of people have private repositories.

Can they contribute their metadata to something and you guys can and do an analysis together so they can at least see what's going on in their own organization. Is that worth doing or? Absolutely.

So that is essentially why Mozilla started theirs first, so they wanted to do an internal test to see if they could improve the quality of code contribution just inside of their organization and that's why that one exists. They did not do a lot of like formal quantitative analysis, but qualitatively was reported that this improved the ability to communicate about issues and pull requests. which I find really interesting and I didn't really expect I wouldn't believe that just hiding that data in that single condition would improve the quality of the conversation that happens afterwards, but I think it does really make sense if you consider that.

When you're just looking at the code and you really initially am making that decision. When I first write up whether or not I'm going to accept it my reasoning for it. I'm not thinking about communicating to that individual.

I'm thinking about communicating at the quality of the problem which allows that information to be better distributed to potentially other problem solvers and better and other Engineers. I think that's a really good way to think about improving our communication. All right.

I think we've proven the point it is a math problem Sal. Thanks for being on the show. Thank you so much.

All right folks. This is our last episode. This has been a blast to be here.

We hope you would all come sometime in the future put it on your travel plans because as great it is to talk to you. It's also awesome to talk to you all in person, but this will have to do for now and we will see you all at the next big event.