Red Hat Brings Enterprise AI at Scale Into Focus

AI Leadership Insight Series. I'm your host, Mike Vizard. Today, we're with Tushar Katarki, who's head of product for Red Hat AI Platforms, and we're having a little chat about, well, how to live with AI.

I think building the models is one thing, but once you install these things in enterprise environments, well, they're not so easy to get along with, and maybe there's some things we need to think about. Tushar, welcome to the show. Thank you.

Thanks for having me. Great to see you, Mike. Walk us through, if you would, what are the challenges that people are running into once they put these things into production environments?

Because I think, as they say, to be forewarned is to be forearmed. Yeah. As you know, the industry spent the past two and a half years proving AI pilots work.

2026 is the year the question is flipping, right? Can you run this securely? Can you run it at scale?

And also next to everything else that you have to run in your IT paradigm. And that's exactly the arc that we have lived. We started with inferencing about a year and a half, two years ago.

We built model-as-a-service on top of it, and now agents are all the rage, and we'll talk about it. At every step, one thing I'll underline is open source models, and open source in general, has gotten better faster than anyone expected. And the only way to keep up is a platform that one can govern and control.

So I will talk to, in terms of the industry trend, pilot to production, we talked a little bit about that. We'll talk a little bit about inferencing and some of the foundational bets we have made with things like vLLM, and vLLM for inferencing, and LLMD. We'll talk a little bit about model-as-a-service and some of the open source models, including the recent Kimi K2, and how this story fits into that.

And finally, I thought we'd talk about agents, and Red Hat's answer with Agent Ops and AI Gateway for control of agents. So does that sound like a good- Yeah ... framework for you?

Well, let's just get started with where are people running the AI model? " And then the cloud era came, and we started moving the data into the cloud, and essentially we were bringing data to the compute. And now I'm starting to wonder if we've come full circle on this conversation, and we're now bringing the compute to where the data resides in the form of the AI model.

Right. Absolutely. That's a good way to think about it.

And as I said, we have lived that, and I have lived that. So what is driving that shift from what is the cloud to, I'll just say, self-managed on-prem or potentially neo clouds and MSPs, et cetera? The most important glaring one, really, and it's been in the news, are actually three, and I touched upon that.

One is tokenomics and token costs. So everybody started, including I started my journey in this with frontier models, but that leaves you with just being a token consumer and really not producing tokens and not being in charge of your own destiny. And to my earlier point, back a few years ago, the open source models were not capable yet, but the open weight models now are very capable.

So that's the first point. The second point really is beyond tokenomics, you want governance and control. We know these models and the systems around that, that you can build are very powerful, but at the same time, they can be pretty risky also for one's business.

And that's the second one. The third one is, relatedly, is increasingly because of the geopolitical situation right now, every country, every region around the world is thinking about what enterprises are thinking. They're also thinking at the macro level, at the country level, or the region level.

And should I be sending data? Should I be reliant on compute that, let's just say, is predominantly sourced in the United States? So that's definitely a concern that we hear outside, both from a data perspective, but also the hardware and the software stack perspective.

And that's where we feel open source plays an important role. And then finally, I think it's the agents. You could think about as chatbots as very simple agents, but now we're talking about long-running agents that can spawn multiple agents to do tasks, can be self-evolving.

So how do you then secure that, especially even with your data, even if you were to repatriate all this on-prem, how do you control that, is definitely part of the story. Yep. I feel like we've just come through, for lack of a better analogy, the mainframe era of AI.

And are we now moving to a more distributed computing model for AI, where we're going to scale horizontally, especially the inference engines? And does that require a different way of thinking about how I manage those inference engines? Yeah.

Definitely. So you can tie it back to how To that story a little more, let me tie it. If you think about I worked for Sun Microsystems a long time ago.

Those were vertically integrated solutions. And then we kind of got into the era of more horizontally scaled everything. Like the servers, rack-mounted servers, COTS servers that you can build software.

The Linux played a big role in that, and the evolution of that for the distributed world was first virtual machine technology, but then followed by containers and Kubernetes. So we are really extending that as saying that can we use the same kind of model for AI? The one twist I will add to that is just like how that played out, there is the frontier model.

There are lots of as-a-model services now, including inferencing. So in some ways, you could argue that that is playing out kind of the cloud. But also there is a lot of pressure, for the reasons that we discussed earlier, to do it self-managed so that you can have more control.

So there is a little bit of change and similar, but also a little bit of change. The only other thing I will add to that is the modern systems coming from NVIDIA, like NVL 72 or the equivalent that came out of AMD, Helion, I think they call it. So they are, I would call hybrid.

They are not purely vertically scaled systems, but they are also. Like those compute trays that they have in there, they have very fast interconnects, they have storage attached to it. And so although each one of them carries a distinct Linux copy, so you could say that it is a distributed system, there's a lot of innovation that companies like NVIDIA and AMD are making to make it a little more vertically integrated.

So that then kind of then goes to this era of next generation. What is a next generation data center? The next generation data center is built on those horizontal servers with fast interconnects, et cetera, but also allows your GPU and compute infrastructure to be much more what I would call configurable.

So what I mean is that if you go back 10 years, 20 years ago, you would rack the systems, you'd wire them up, and that's fixed. So you cannot take that rack and a server in there and give it to one tenant and give another one to a different tenant and swap them around that easily. And that's kind of where the next generation data center is going.

And so that aspect is new, I would say. Red Hat's been investing heavily in a distributed computing platform and contributed some of that back to the open source community. Describe what it is that that does and the relationship between the various components, if you would.

Yeah. So fundamentally, you think about what is so fundamental to the new AI era, those are models. And the models are really capable.

They can do many things that they could never have dreamed of before. But they need specialized hardware, and those are called AI accelerators in general, but also called GPUs. Sometimes we use them interchangeably.

So we need something to connect those models to the different kinds of hardware accelerators, and that's where the inferencing engine comes in, and that's where we have invested heavily in an open source engine called vLLM, inference engine. And that has become the de facto inference engine, so much so that it is the most popular, if not the most popular inference engine from GitHub stars and number of contributors perspective. So vLLM becomes the core.

It can take any model and it can make it run on any hardware accelerator, be it from NVIDIA, be it from AMD, be it Google TPUs, or even some newer ones. And the analogy there really is think about the Linux kernel. What did Linux kernel do 20 years ago?

It basically abstracted the ISVs and the software that is going to run and the underlying horizontal servers, the x86 servers that were coming to the market, so that, basically, if I am an IT shop that is trying to procure and install and make available services based on a piece of software, I don't really have to think about the hardware. Obviously, I'm going to go and fetch hardware, but I could swap out one server with another server very easily, and the Linux kernel did that. That's exactly what vLLM is doing.

It is saying you can bring any open weight model, you can buy any hardware accelerator, and we will make sure that that model runs, or we, as in vLLM, will make sure that that model will run most efficiently on that hardware accelerator. And efficiency is really more important, even more so than the world of just AppSec and Linux servers in the modern era, because the GPUs are very expensive, so efficiency really matters the most. So now if you build on top of that, there is distributed inferencing.

So again, these different servers, you could think about them as isolated vLLM running on GPUs. But the challenge with that is While VLM does an amazing job for one model to one GPU at a time, when you have many models to run, some small, some large, some which are audio models, some that are text models, some that are video models for different input, what I'll just call multi-modality. In such a scenario, when you have a bunch of models, bunch of users, and agents that have different needs, and different, by the way, SLAs that they want, right?

Like if it is a human involved, then the latency becomes important. If it's an agent, then, potentially, a throughput becomes important. And then you have a bunch of hybrid infrastructure anyways, multiple accelerators.

So how do you run multiple models and match that with the available hardware that you have or you can procure? I think that becomes the challenge for a distributed system, and that's where LLMD comes into the picture. The big thing with LLMD is when you have a more general distributed need for multiple models on multiple hardware infrastructure, potentially even crossing boundaries of the rack or the data center or even a region, then you need something that can do that and that's distributed.

That's where the distributed part comes in, that's LLMD. I'll pause there. We can go next to the model-as-a-service layer.

But do you have anything there before we get- Sure, yeah. So how does this evolve is the way I think about managing my models and to your phrase, model-as-a-service. But it also seems to me, are we getting closer to the point where there are different types of AI models, some of them are smaller, more specialized, and we need to figure out how to manage that and distribute them a little bit to the right place at the right time, and this is becoming the art of the thing?

Yes. And actually, that's a good point. So one of the big things with LLMD really is what you just alluded to.

I have a bunch of models. I have a bunch of SLAs that I need to meet, but I do not want to think about it as a platform admin or a platform provider. So that's where LLMD has that intelligence, which is specifically meant for AI workloads.

And I'll dig one level deeper, and I'll explain what that means. So one is, AI workloads are very bursty in nature, even more so because you can imagine the prompts being very small and the output being very large, in the case of some kind of a generative use case where, or you can imagine the prompts being very large or the output being very small in some kind of a summarization use case. Or there could be what are known as multi-turn.

You are having a conversation or an agent is trying to do multiple things. So some of that is repetitive. So long story short, LLMD understands the particular characteristics of a AI workload that I just described and optimizes that for the hardware infrastructure, and we can talk about things like KV cache, prefix-aware cache, or KV cache-aware routing to different models.

We can talk about prefill and decode disaggregation and what efficiencies they bring. We can talk about caching and hierarchical storage. You have a lot of context building up.

There is no space in the GPU memory. Can I offload it? Similar to CPU, to cache, to memory paradigm that is bringing to the world of use.

That's what LLMD enables. And then finally, you have these large models like Kimi K2. Very good model, but it's a very large model.

I cannot run it on one or two models. How can I run it, the whole model, and on a distributed set of GPUs because it won't fit in one GPU? So those are all things that distributed LLMD enables.

Now, layering on top of that, now LLMD, as powerful as it is, you still need a governance mechanism above that. And by that I mean, just like how I would have gone to a OpenAI and, as a user, I could have said that, "Hey, give me a key. I want a certain model that I want to serve, I want to use," and hit the APIs for that model with my API key.

That's the kind of service that you need to provide. But beyond that, you need to be able to see how many tokens I consumed. You need to be able to do some kind of a chargeback or a showback.

You need to do some kind of a rate limiting, quota management, et cetera. So that's what the model as a service layer brings. And then if you expand that a little more, okay, it doesn't stop there.

But beyond that, I also want to make sure that there are input and output guardrails to this model. I want to go beyond that and make sure that the models can do tool calls because that's what the agents require. They need to be able to do tool calls.

So that basically the model-as-a-service is evolving into this, what is known as the AI gateway layer, and there's a lot of talk in the industry about it, but we certainly have also evolved our platforms in that direction. So a lot of this comes under the heading of tuning and optimization, and my question is: Am I, as the IT team, going to have to have those skills or as a model as a service platform going to embed those into things? And I may just describe my ultimate intent, but all these things are kind of automated.

Yeah, exactly. That's the intent, that's the goal, that's the intent, that's the target. Now we are in different phases of evolution, right, of that technology.

So but that's the intent, right? Like I as a platform provider, should be able to give my consumers, my customers, whether internal or external, the best and the latest models. I want to be able to provide the SLAs that they require.

Some may require latency SLAs, some may require throughput SLAs. I should be able to control that. And as I said, I should be able to do this safely and securely.

And to your point, I do not have to think about this. I want the underlying infrastructure software, the AI infrastructure software that we are describing here to take care of this with some control that I want to express, right? Like that's the SLO, lever that I want, right?

I want to be able to say that these workloads have more priority, they are more latency sensitive. These other workloads are much more batch oriented. Or I want to be able to say, "Hey, look, based on the prompt, this looks like requires a lot of thinking and reasoning.

" Whereas this other task, something can be done by a specialized model or even a small model. And that kind of intelligent routing or what is also referred to as semantic routing, that's definitely part of that. And the last one I'll say is cost-based.

Definitely customers are looking for, basically my intent is to optimize my cost, obviously, and therefore, why don't you take care of that problem for me? That's where the cost-based routing in that gateway layer comes into picture. All right.

So ultimately, what's your best advice then to IT teams about how they should think about their role in all this? Because it almost sounds like they're model service providers. Yeah.

I think model service providers and agent service providers are kind of couple of key constructs to think about. So, IT teams should be thinking about, what does that stack look like? What are the pieces of that stack that I need, and how do I, obviously provide the best service that I can with the budget that I have?

But, and you can talk about budget in terms of actual dollars or you can talk in terms of compute infrastructure or a mix of those two. Given what I have and given my short, midterm, and long-term projections, what can I enable? And then beyond that, both for models and for agents.

And then like I said, I think the most important thing really also is beyond that, right? That's the service that they are providing, that's what the consumers are using, but how do I do it in a secure and safe manner, and so that I can have control over it, I think is the other key thing that they should be thinking about. All right.

Well, folks, you heard it here. Hey, there's a lot to think about and there's a lot going on with AI, but the more you think about it, the more it looks like just a new class of workload that still has to be managed by IT folks. Just we need a smarter way to do it.

Tushar, thanks for being on the show. Thank you, Mike. Nice talking to you.

All right. ai Leadership Insight Series. You can find this episode and others on our website.

We invite you to check those all out. Until then, we'll see you next time.