Erik Erlandson, Red Hat | OSS North America 2023

This is Techstrong tv. Welcome back to the Open Source Summit. We're here with Eric Linson from Red Hat, and we're talking about Climate OS and data science, and it's a big project, and a lot of work is going into this, but we understand that there's definitely an effort to collect the data, but once we collect the data, what are the data science challenges?

What's required? And ultimately, is this gonna lead to some form of AI being applied to all of this? What should we expect?

That's a great question. Um, truthfully, I think an awful lot of the value comes simply from, uh, properly, uh, federating the data. And so at the platform level, um, almost all of our efforts so far has been building out an open source data mesh platform, um, that allows people from the community and potentially all over the world to come to the community and offer new data products, um, for use by everybody.

Mm-hmm. Um, and so on that level, probably 90% of the value is there. Um, however, there are definitely some data science, um, efforts going on.

Um, one of the, uh, important channels of, uh, climate reporting information is just the reports that different companies file publicly. And this data is publicly available. Harvard is all generally filed as unstructured documents such as pdf, and so there is no unified way to, uh, transform this data into something that's, you know, organized and tabular, that normal algorithmics can operate that all the time.

So here is where, um, a lot of the newest technology and large language models such as chat, G B T, um, actually may play a role. You can imagine training one of these models specifically to be optimized for, you know, extracting all of the different reporting numbers from a document regardless of how it was actually organized. And in this case, your program is actually literally a series of questions or, you know, chat G p T style prompts.

Um, they say like, find me the, uh, the reported value for CO2 emissions for company X, and you can simply collect as many of these questions as you want, run them all against each document. And you know, with some, with some, you know, possible error rate, you'll get good answers. And any errors that are made can be generally fixed later by, uh, you know, human humans in the loop or after the fact.

It almost sounds like part of the magic of this is that the data mesh is not pulling all the data into some central repository. It's more like the data becomes almost like code and I'm calling that data or as I need it per se. And is that a more efficient approach to this challenge?

Um, it, it is more efficient. Um, and perhaps more importantly, it's really the only way you can scale this kind of open community. You have to make it, you have to make it as self-service as possible, both from the point of view of people from the community who want to provide data, um, but also the people coming from the other side who want to use it.

And so like we're in, we're in, in the process of implementing, um, a component we're calling the data exchange, which allows anybody out in the world to approach us and register a new data product. Um, and it becomes a searchable panel on the data exchange. And that means that people on the using side, the consumption side can come to this, find pieces of data, understand their descriptions, what their schema are, um, whether it's, you know, coming to them via a, uh, SQL query or REST API or Roth file.

Um, and so this is a hugely important in terms of being able to just scale the governance and the operations of the community. The data itself may not be accurate or inaccurate depending on who created it in the first place, but will this make it easier for somebody in a particular vertical industry to kind of do an analysis of what is the impact of carbon on aerospace per se? Because the data will just be easier to find?

That's correct. Um, one of the reasons we wanted to create this, um, data commons as we're calling it, is because currently only the largest companies can actually afford the resources to assemble this data for themselves. And that leaves, you know, thousands and thousands of smaller institutions out in the cold.

They could never, you know, create this. But by, you know, creating an open data commons, it's like we can share the data. So once somebody, once somebody surfaces a new kind of data stream, for instance, um, you know, the relation ownership relations between companies or new climate models or simply the locations, the physical locations of actual corporate assets, um, then it becomes potentially available to like the entire world to use.

Will this be applied for other things besides climate research? I mean, there's other large data driven project, so is this something that becomes a test case for other things? Uh, it absolutely is.

In fact, um, you know, we are designing it very explicitly as an open platform, um, both in terms of the open source components, but also the open source deployment knowledge so that anybody can come from the, the world, get onto GitHub and see exactly how it is we're deploying our components onto the cluster, um, what versions we're using, the actual source codes that are doing computations. Um, all of it is available for people to use. And if you wanted to come use our data commons as a reference architecture and make whatever modifications you wanted, um, you could use our model to deploy your own version and fill it with the data that you want.

How hard is it to set all this up to I have to be a rocket scientist or can I be a reasonably mere immortal IT professional? Um, I think an anybody who has facility, um, doing like DevOps kinds of operations, like if you're, if you have the basic skills to do deployments on something like a Kubernetes cluster, you should, you should be able to take what we're doing and, you know, replicate it. Mm-hmm.

In fact, we recently, um, one of the, one of the community members recently published a fairly detailed set of instructions on how he deployed, exactly how he deployed, uh, all the components from the cluster on up, um, to GitHub, which is also publicly available. Mm-hmm. Can I program this so that anytime there's an update to a particular data source that it will tell me or alert me somehow or other because, um, the research is continuous, right?

That is correct. Um, we, we, that is on our roadmap. I, uh, we do not have like alerting capabilities yet, but we absolutely intend to for exactly that reason.

Um, we are designing most of our ingest, um, as data, as code style pipelines, um, using things like either Jupyter eil or things like D B T and running on airflow. And so it is a true data as code operation. And of course, as with all the such kinds of pipelines, you can configure them to trigger of data on the input side changes.

And eventually we would like to have it so that anytime data is updated, people can subscribe to alerts for that. So historically I would've had to what, hire a dozen data engineers to go do this kind of thing or, um, and now it's basically becoming something that's more automated and democratized. Is that the end result?

That is the end result? Um, I think, you know, in a sense it's just like open source where it's like, in fact, you are taking advantage of lots of people's, um, you know, collaborative time, um, on the data side, it's going to be the same. People will be able to take advantage of, you know, thousands of, you know, data engineer hours in terms of individuals, you know, collectively.

So what does that community need right now? I mean, we have folks that are watching this. What exactly are you looking for and what kind of help do you need?

Um, we can always use people who do have, um, you know, DevOps knowledge of how to deploy things in clusters. We are always looking for community members who would like to offer new kinds of data products into the system. And either, either of those two things would be extremely valuable.

Is this something that, uh, the average student can help with or is this, do I need to have those kind of certifications and high level skills? I mean, what is the continuum people that can help from this kind of project? Um, we can use people at any skill level.

I think anybody who maybe doesn't have high skills in all these areas but is willing to learn, um, and contribute their time could be extremely valuable. We're very happy to try and help spin people up. Um, as I said, all the code that we have and all the deployment, the get ops deployments we have are out there for them to learn from.

So this is a place where you could finally get some real world experience for free and kind of do something that you know will help your career. Yes, it would, I imagine look quite good on a resume and you would know that you were actually helping, you know, the entire planet align all of our human economic activity, um, with trying to control our environmental impacts. So what's the hard part about all this?

I mean, it sounds like, you know, it's all set and ready to go. So what's the piece that's missing? Well, one of the pieces that's missing is it does take a lot of labor to figure out how to deploy something.

Um, once you have a deployment, you know, the knowledge is embodied essentially out on GitHub, but it takes people, you know, the time and effort to do the, um, experimentation and development to make it all work. Uh, a lot of the challenges that we have had are integration challenges. Um, we are using tools like Trino, which has many, many data connectors, but not all of them work perfectly.

And it takes effort to like actually take, take these connectors, get them working file bug reports upstream if necessary, um, things like, uh, Linux Foundations, open metadata, um, you know, getting that to integrate properly with things like Trino and our, uh, vault single sign on, um, you know, takes, takes experimentation and knowhow. So what's your timeline for this project? I mean, certain aspects are ready to go and other things are work in progress.

So when should we expect to be able to say, Hey, you know, we're seeing a positive impact and these are the results? Uh, that's a great question. We're hoping to, uh, we are, we are in the process of migrating from a legacy cluster to a new cluster.

So as I speak to you now, a lot of, a lot of this is shifting under our feet. Um, however, the result of that shift is going to be a substantially, uh, faster development cycle. And so we're hoping to have, uh, accelerated deployments on a lot of the initial applications, um, by the, you know, one or two quarters.

If that's the case, then am I gonna see like a bunch of climate products for that are based on this? I mean, is there gonna be a whole ecosystem of vendors around this thing selling what dashboards on the data or how do you envision that coming to be? Um, that's actually a very interesting question.

We know that we expect many of the users of the data to be using, uh, the data internally. And so like if I am, uh, a financial institution who manages things like mutual funds or other kinds of investment portfolios, um, they will be taking advantage of our data to, you know, improve the quality of their products and make sure that their products are aligned with various, um, ESG initiatives and environmental, social and governance, you know, per criteria. We see people selling carbon credits these days and getting, um, uh, value out of how much they reduce their carbon emission.

Ultimately, that activity needs to be driven by some sort of data that confirms who has what, when and where. So is that part of the, how we're gonna make that whole model more accessible to people is by making that data more, um, simpler to find and then equate it to some sort of carbon credit? Yes, that's exactly correct.

Uh, again, as I mentioned, it was, you know, if you imagine maintaining publicly available databases of the, you know, physical latitude, longitude, locations of actual corporate assets, um, so that you can correlate it with potential climate impacts, um, anybody can then use that information to, you know, align your own investments or create other kinds of climate risk reporting. Uh, there's enormous numbers of applications, um, ones that I'm sure that I haven't even thought of yet, and we're hoping community members can come to us with those. And there's a lot of hidden dependencies in the relationships between things.

So an example that we may all know is we may use a cloud service per se, but um, that's creating some level of carbon impact, but we don't really know on our side because we don't have the report of the data. So is that part of this whole process is to kind of say, and here are all your carbon dependencies? Absolutely.

In fact, what you are describing are called the scope one, scope two and scope three, uh, emissions and scope one is simply, literally like what I emit as a company. For instance, if I'm a, if I'm a factory and I'm burning coal, that's my scope one emission. Um, if I'm a, if I'm a software vendor and I'm running my tooling on something like AWS or gcp, um, the carbon emissions of the stuff running on the cloud are scope two emissions.

I'm hiring somebody else to burn energy for me. Um, and even more, more complex, the scope three emissions, if I am say a factory, I might have, you know, supply chain coming into me. Um, and the scope three emissions include like all the emissions from all the people who supplied parts to me, and furthermore, the, you know, emissions of anybody who I saw my parts to.

Mm-hmm. So in order to compute scope two and three emissions requires a very comprehensive knowledge of what companies own, which other companies, but also what companies they are buying or selling things from. And so there are like enormous, um, you know, data engineering challenges, uh, for assembling that kind of data.

Do you think we'll get to the point where I could have a, what if dial on all this data and I could start tweaking and if I change this, this will be the impact on the carbon consumption rate and people will be able to make more projections about what change might? Absolutely. Um, in fact, we already have prototype applications that allow you to compute, um, the physical hazard models for various kinds of hazards.

Things like, uh, work loss from excessive heat to, um, you know, flooding hazards from increased rainfall and you know, forest players, all of that. And you can just pick, you cannot just pick the actual hazard you're looking at, but you're also looking at different predictive models. So some models are more pessimistic than others.

Um, what it looks like depends on how far out you look, you can choose to say, I wanna look at what might be happening in 2030 or 2050. And so all of these are parameters that we are already, um, supporting in our physical models. And we're in the middle right now of wiring all that up to things like the actual physical locations of corporate assets.

And once we have that wiring complete, um, it'll become an, a super powerful tool for like actually connecting physical scenarios with value impacts. Mm-hmm. Will we get to the point where I'll be able to look at some sort of rating and assess somebody's carbon consumptions or emissions and decide whether or not I want to do business with them based on that?

Because we're already seeing a lot of companies, especially in Europe where people are saying, yeah, I don't want to buy from this one, if that one has, you know, a negative impact on the environment. Yes, absolutely. And as you know, you can even now or purchase, um, things like mutual funds that focus on sustainable only companies.

Yes. Um, and really part of the goal of the OS climate initiative is to make that kind of data available to every financial institution. So that ability isn't just the, you know, purview of very large corporations, but anybody who's selling it or even individuals, There's a lot of emotion in this issue, is ultimately the goal is to, you know, take the emotion out of it and just make it about data and the science.

Yes, it's all about being data driven. Um, and of course you can never, you can never take out the societal or individual emotional reaction to what that data says, but you can at least make it objective and say however you feel about it, this is the projected impact of this investment. Alright.

What is your best advice to companies that are interested in this? How do they kind of, how should they think about it? Because a lot of folks you talk to, they find it a little overwhelming and they find it kind of, uh, you know, the data involved, it's kind of challenging and they just don't get started.

So where's the point of departure? org. Um, that would be a great place to start.

We do have, um, an OS climate GitHub organization, which you can actually look at the code we're developing, but it also that those repositories also contain many pieces of contact information, architecture, diagrams, governance, um, you know, meeting times the, the usual open community kinds of resources. All right. Cool.

Folks, if you want to help save the planet, this is the man to talk to. Hey, Eric, thanks for coming by. Thanks Very much for having me.

All Right. And we'll be back in a minute.