Defining True Open Source AI with Red Hat’s Steve Watt

ai video series. I'm your host Mike Bazar. Today we're with Steve Watt, who's a distinguished engineer in the office of the CTO and Red Hat.

And we're talking about AI weights because, well, there's a lot of debate going on about what constitutes an open AI model, but Steve's gonna explain. Steve, how you doing? Good, thank you.

I appreciate you having me here. So I feel a little bit like deja vu. There's this whole debate about what's open versus open source.

And a lot of it seems to come down to the issue of, well, how accessible and transparent are the weights in an AI model? But you know more about this than I do. What is the issue here?

Yeah, so I think the issue is that, um, the term open source AI is being used rather indiscriminately every time a model is released publicly. And after a bit of time, you know, the industry sort of come to the point of, uh, well what does that actually mean? Um, and especially in contrasted with, you know, the way the industry understands open source software, um, I guess, um, some of the licenses and how restrictive they are, um, doesn't seem to gel so well with our, our understanding of, uh, open source software.

And so, um, you know, it's definitely fostered a debate in the industry. There's a couple of different, um, uh, industry, uh, organizations that have come up to try and define it. But, um, you know, I think it's, we're still sort of settling in on what that actual definition is.

So if I'm an enterprise and I'm looking at all these licenses, what should I be on guard for? Because what might trip me up down the road? Yeah, I think it, it comes down to what your standard expectation of an open source license is.

And, um, the first, if you actually look at the OSI definition of what open source is, I think it's number five and number six, there's like 10 steps. Um, but you know, the license, um, has gotta not discriminate against persons or groups, um, and typically doesn't limit use, um, um, around fields of endeavor. Um, and so that, that is like the first thing that I would look at, um, is just how restrictive, you know, Apache too, if it's Apache two license, that's something we all understand pretty well in the industry.

Um, uh, both I think during the big data or scale out era and the cloud native era, everyone got real familiar with Apache two, but, um, there's a, a couple of new ones that have come out that are a little more restrictive, especially if you're based in the eu. So I think that's like the first thing that, that I would look at. You know, and if it's Apache two, it's something you recognize and understand, fine.

If it's something different, maybe dig in a little more. Mm-hmm. So it, it almost comes down to me at least how permissive is the license, including the weights.

'cause maybe a lot of the models aren't useful to me unless I get the weights to go with it. Right. Well, and it's, yes, that's correct.

Um, but, um, here's also how much, it's weird because I think in open source software licenses are pretty well understood. Um, you, you, you tend to, it, it tends to be pretty easy to identify if derivative works, um, and if the license on that is on the up and up. But with AI and models, which you can have is imagine a scenario where you have a restrictively licensed model, someone downloads that from hugging face, fine tunes it, and then publishes it with an Apache to license on hugging face.

How would you know like that they've, they've obviously taken something that was restrictive and then repurposed it, like made some minor changes to it that violate the license and the sharing, but then they switched the license and the next person that comes down and downloads that derivative product, they'd have to do a bit of reverse engineering to actually understand what has happened in the past. And, you know, they're actually violating the original license. So there's this weird, um, onus that that's at least right now being pushed onto the user to actually figure out if the actual person presenting the license is doing it correctly.

So Is it me, but I feel like we're also having another deja vu experience where the proprietary AI models seem to have jumped out to an early lead and they kind of drove all this, but now the open source models are catching up and I feel like this whole cycle is happening maybe a lot faster than it's happened in the past. But what's your assessment of where are we? I totally agree.

Um, I feel like, um, we're on a treadmill and each week someone's increasing the speed, you know, the, um, and uh, I think this is, you know, part of the nature of open source, which is it creates a steel sharpened steel effect, steel sharpened steel effect, where, um, in the same way that enterprise open source eventually caught up and started dominating where the new ideas were coming from in enterprise software, it's just, like you said, started proprietary and open source is caught up and is now dominating the ideas. And you have this effect, um, where if I publish my model and then I publish the paper of the decisions I made to actually produce that level of performance for the model, and it's all fully open, the next group that's trying to improve on your idea can take those ideas and improve on them. And, and that's the, the classic access and transparency and agency that you get with, with open source.

And so it's rapidly increasing the, the rate of the innovation. I think that's a pro. I think the, the challenge, um, especially with large language models is the high barrier to entry, right?

Which is you have to be able to pay for the infrastructure, right? Which is millions and millions of dollars. So it's not quite like open source software where a college student in their mom's basement can, you know, come up with the latest coolest open source software project where the barrier to entry is extremely low.

You know, it's a laptop and a GitHub account or something to that effect. You know, where to produce a, you know, a trillion parameter, a multi-billion parameter model, you, you've gotta be able to, it's financially prohibitive, right? So I I think it's sort of a mixture of both where if you can afford to get into the large language model game, the open source is increasing the speed of the innovation.

Do you think we'll see a lot more smaller models? Because, you know, I hear people are using techniques like distilling to kind of get a jump up on the next model. And maybe what we're gonna see is a proliferation of a lot of smaller open source models.

I absolutely do. Um, I think, um, nobody really talks about the edge market that much these days, but I think that's getting that model closer to the end user, that last mile, um, is gonna, uh, for the same reasons people run applications on the far edge, which is it provides a better user experience, it's more responsive. Um, I, I think that that's one very interesting use case long term for smaller models.

Um, the other thing is, um, you know, purpose-built models, you know, not, not everybody's application needs to know who won Wimbledon in 1995, you know? Um, and so there is, you know, a foundation model or a large language model, like there's a former Wikipedia, right? Uh, it knows everything about everything, you know, and, um, there are just a lot of applications don't need that.

So I think when you get purposeful models, I think that's, that's when application, um, there's a whole flavor of this that it's, that's kind of interesting. It's related to cost though, which is like quantization and ification of larger models to make them smaller because you don't have the, the compute infrastructure, uh, or you don't want to pay for the compute infrastructure to run them natively as they're presented. So if you can actually compress them down to run into smaller memory footprints, it's much more affordable, uh, for you to be able to run these, um, maybe within your enterprise.

And there's another aspect of this, which is scarcity of, uh, compute resources in that you just can't buy all the, the compute that you need, um, or maybe you can't afford to. And so there's an interesting trend around smaller models that I would, that is related to smaller models, which is, you know, using what you got. So it might be a couple of Nvidia one hundreds that are older, you know, and maybe you had a couple of H 100 GPUs or something, but you've got to, you don't have all the H one hundreds, you need to run this thing 'cause so can you compress that thing down to actually run on what you got?

And that's also a driver towards smaller models. And I can't help but wonder if the smaller models might be more accurate because there's a limited amount of data. So I'm not trying to boil the ocean with a lot of data that might be inaccurate.

Certainly. And I think also just in, uh, with supervised learning, um, and just validating the outcomes. If it's more fit for purpose, I think it's a little easier because there isn't such a, you know, you're not, you're not testing the validity of how it understands the world's knowledge.

It's just, you know, one domain, just tennis, you know, or something to that effect. So As we go forward, I mean, a lot of enterprises are trying to figure out how to operationalize all this stuff, and, uh, they all seem to have got a lot of experiments going because that doesn't cost so much. But if the infrastructure ultimately is expensive, do we need to kind of start picking some winners here and maybe make a handful of bets about things we're gonna actually put into production?

Yes. Um, if I'm interpreting your question correctly, um, I think what you're saying is that organizations need to, or, or because of infrastructure scarcity, having to be judicious about what they do deploy and run. Um, yeah, uh, it's, it's absolutely true.

Um, and, uh, we're seeing that already. Um, the, the, there is an interesting tangent to this though, which is, you know, creating optionality in your runtime for your AI stack. So just to nerd out for a second over here, like there's a, a pretty interesting open source project called Triton, which is basically part of the broader PyTorch ecosystem, which is, you know, the core runtime where you can do pre-training or model inferencing or fine tuning on it.

But, but what this does is it creates this hardware abstraction layer across the different world of GPUs. And so, um, it allows you to sort of write your application logic one way and then basically be able to light up a variety of different kinds of hardware. And I think that is creating some more options for people to, um, as I sort of mentioned earlier, that trend of like using what you've got, right?

Um, it, it sort of expands the ecosystem and allows you to, um, in some cases, here's what you've got or here's what you can buy, what you can find, and then run that on your infrastructure. Are you hopeful that maybe we'll see additional classes of processors that'll be used for not everything is a training exercise. I might be able to use something for inference, so maybe I don't need as many GPUs and there's other things to play with and we should start thinking about mixing and matching things.

Absolutely. Um, I think this is to me, one of the most exciting spaces in AI right now, which is from a compute infrastructure, we're in the wild west. And what I mean by that is, um, Moore's law coming to an end, okay?

Or some may argue has ended. Um, so models can't get the performance they need from the CPU, well, primarily because it's not parallel enough, doesn't offer enough cores by, its several orders of magnitude than what a GPU uses. However, um, so even if you're, you're going to a GPU, A GPU is still a general purpose graphical processing unit.

It works, it's designed to videos, video games, video drivers, it's, there's some applications towards cryptocurrencies and then there's ai. And the fact that it's trying to address all these different use cases means that it's not particularly energy efficient, which has caused this whole birth of this, these accelerators that are purpose-built for ai, just like you said, um, like rocks, um, accelerators just for inferencing, right? Um, and, um, being able to light all of this stuff up in the core runtime.

So for us, that would be red, a enterprise Linux, like the operating system so that your, your software runtime that's providing the instruction set to the accelerator, um, knows how to talk to that thing. The drivers exist, the kernels exist to be able to do that, that, um, has become much, much more complicated than prior to the explosion of gen ai where all we really had to do in the operating system was light up X 86 and light up arm, you know? And, um, and so now we have, you know, I was looking at a, a new accelerator the other day and, you know, we're used to, you know, needing a microscope to look at processors.

This thing was this big, which was huge, you know, and so it's a, it's a, um, uh, so if this is not video, just audio, I'm describing something like the length of my forearm squared, you know, um, and um, the trying to do all of that enablement in the operating system so that these things can get hued, um, is a significant engineering challenge, you know, for the whole open source community. The other thing I'm scratching my head about a little bit is to your point about the user experience, more and more of these models are gonna have to get closer to the point where the data is created and consumed, and that pushes it out to the edge where the hardware is, uh, limited in its abilities. And if I'm pushing those models out to so many edge devices, it may get prohibitively expensive to use something that's a commercial license.

So is that whole part of the model and the whole part of the framework, uh, at the edge gonna be inherently open source? 'cause it's the only way I can really afford to make that work? Um, that's a great question.

I will say, um, we did reason through this when the world went up from scale up to scale out. So, you know, when we, when we went into di distributed systems, there was sort of this question being raised as well, where instead of just buying a bigger and bigger database, you bought, uh, a, um, sorry. When you, when you needed resource requirements, you just buy a bigger server so that you could run a bigger database as opposed to buying, you know, 500 et cetera.

And Edge is just sort of an extreme version of that. Um, we, uh, have built, we have an open source project called Micro Shift, which is a very, very lightweight, paired down version of OpenShift. And um, and it allows you to, um, run on an edge device like an Nvidia Jetson or something to that effect.

Um, and then you can pull an operating system through, um, and then you can actually deploy and lifecycle your applications on top of this. So, um, we, you know, if you're, let's just say, um, forest, if scenarios aren't always sort of commodity infrastructure, so for example, sometimes, um, if you're your age is a, a drone that's survey doing like land surveying, you know, like flying through valleys and mapping them and things like that. Um, there's a commercial use case where people buy the downstream flavor of that product called Red Hat Device Edge to be able to do those things.

When it comes to ai, the question is though, you know, it's more than just OpenShift. It's like the whole inferencing stack, right? And can we get that inferencing stack small enough to actually run on that NVIDIA Jets and or whatever they're running inside, um, the Forage device.

And that's an area we're actually exploring in my group right now. And, um, you know, we've just kicked off a project on inferencing on the forage to, to better understand this. Um, but it, but it is just like you said, it's trying to fit in that resource budget.

So what is that One or two things that you see organizations who are getting all this right, doing that you kinda wish other organizations would follow suit? Yeah, so, uh, well on a couple of dimensions, um, I'll start with the simple one, which is, you know, if you're applying ai, um, so I have a team that, that looks at our software engineering processes or just our general processes inside Red Hat and looks to leverage inferencing to improve automation, reduce, reduce, reduce engineering toil. We've built a bunch of really great solutions that work well.

The challenge is adoption, like, it's sort of like a solution without a problem, you know, and we've, um, we've found that another class of problems where we, when I find that if we're actually able to run into an engineering team that says, oh boy, if you could just do this for us, um, as opposed to us automating in the white space, like if we could just solve that problem for 'em, um, we've had a lot of success there. And then expanding how, what we're building to broader and broader and richer use cases. So I would just say start with the problem.

Although that's a pretty well known thing in our industry that sometimes we just forget when we're automating with ai. Um, and I think, um, yeah, I think that's the main one. You know, I would go with that if, um, it's, it's the, uh, being the most effective for us.

All right, folks, some things stay the same no matter what era it is. And yeah, the more we focus on an actual problem that somebody wants to solve versus just science for science sake, the more useful everything gets. Hey, Steve, thanks for being on the show.

My pleasure. All right. And thank you for all watching the latest episode of the Techstrong AI video series.

You can find this episode and others on our website, and we invite you to check them all out. Until then, we'll see you next time.