CloudBolt COO Yasmin Rajabi on the Challenges of Deploying AI Workloads on Kubernetes

Hey guys. Thanks for the throw. We're here with Yasmin Rabi, who is COO of Cloud Vault, and we're having a little chat about, well, whether or not Kubernetes is the de facto default platform for AI or not.

Yasmin, welcome to the show. Thanks for having me. We have seen at least early on that the hyperscalers pretty much use Kubernetes extensively, but in the enterprise, things may be a little bit different.

Some folks are using Kubernetes, some folks are using other platforms. What are you guys seeing? Um, I mean, for us, obviously we're biased 'cause we get involved if an organization is, uh, using Kubernetes.

And I think what I see the most is that across every enterprise there's at least a little bit of Kubernetes. It depends on whether there's a lot or, um, just, you know, in pockets. Uh, where we see the most Kubernetes is like fintechs.

Uh, large banks are mostly deployed on Kubernetes, and then a lot of hosting platforms as well. When you have multiple copies of the same app that needs to scale differently for each customer, uh, we'll see a lot of those types of organizations on Kubernetes as well. Yeah, in theory we will AI drive more adoption of Kubernetes because from what I can see, there's a lot of data processing and that needs to scale up and scale back down.

And it seems like Kubernetes is the natural orchestration engine for that. But are there other ways of skinning that cat? Um, there's definitely other ways of skinning the cat, but what I have seen a lot is, uh, for example, spark jobs on Kubernetes has been exploding.

Uh, 'cause people have a lot of the data jobs they run for like from anywhere from five seconds to five minutes. Um, and whether or not they should be on Kubernetes, we're seeing a lot of that on Kubernetes. Uh, and I do think it's driving the footprint in the organizations as well.

Mm-hmm. Is there gonna wind up being a Kubernetes skill shortage as a result? Most likely.

I mean, we kind of are at one, uh, even now. Uh, a lot of the organizations I talked to, they'll have like one or two experts and then everyone else is still in that early learning journey. Um, even most recently, the CNCF came out with a, like a Kubernetes AI conformance document, um, where they're starting to lay out what, what are the best practices for being, uh, AI conformant on Kubernetes.

Um, and even just to get there, like you need to be Kubernetes conformant, there's some standards and things you need to hit, but, uh, what they launched was essentially a way for people to do their own kind of self-assessment of do I have the must have practices in place? Do I have the should haves? Well, obviously like we'll see that evolve, uh, over time in the next year.

But, uh, it's giving people a place to start because I think a lot of folks were like, I don't know what to do. Like, the question we get a ton is, okay, I'm starting to do AI on Kubernetes, but like, how do I optimize this? Where do I start?

And, and most folks don't know where to start. I think part of the issue with Kubernetes has always been that there's too many knobs to turn. So can we just manage this at a higher level of abstraction where mere mortals can do this and not everybody has to be a certified Kubernetes engineer.

That's a lot of what we try and focus on is what are the pieces, what are the knobs that maybe you don't need to tune and you can let a machine tune. Uh, it's funny 'cause we're, we're an ML company and often that gets grouped into like AI ml and it's like, oh, do you use AI to then optimize ai? Uh, what I like to tell people is like, what we're doing is just advanced math.

And if you can use math to solve problems that humans don't need to, for example, configuring Kubernetes. So how do you set your request? How do you set your limits?

How do you do that for jobs that only stick around for a short amount of time, but when you do them at scale, that starts to take up resources. That type of abstraction for humans, kind of take some of that toil out of it. And then the, the thing that has, uh, helped us for decades now is just automation.

So rather than having a human go manually configure things, deploy things more with a push button style is just have the automation run it, um, and, uh, run at the intervals that make sense to the organization, but use automation to solve the problem. We've been having this conversation about stateful versus stateless on Kubernetes for as long as I can remember. Does AI kind of result in a lot more stateful applications running on Kubernetes itself, or is the data being accessed externally?

Um, I think I, I'll say it's, I've seen a mix of both. Uh, we still are seeing a lot of stateful apps, uh, on Kubernetes. And, uh, whether or not people are addressing the best practices, I think comes to the maturity of the organization.

And to your previous point, the skill sets they have in Kubernetes. Um, I think no matter what the technology is, there's always the right way to do it and not everybody follows the right way to do it. So you kind of have to build the tooling around anyway that a, that an organization might choose to do it.

Um, as you kind of think all this through for a minute, where are you seeing Kubernetes? Is it all up in the cloud or are people deploying it on premise? Because a lot of these AI apps are accessing data that already exists locally and maybe, I don't know, just data gravity just driving this whole decision?

That's a great question. Honestly, across our, uh, customers, it's probably a split. Maybe it's like 60 40 public cloud to on-prem.

Um, but we, uh, a lot of folks are doing Kubernetes on-prem and they'll either do vanilla Kubernetes or OpenShift is, uh, I'd say in the last six months I've heard way more OpenShift than I have probably in the last two years. I, I think the, uh, migration off of VMware is, OpenShift is a nice, uh, first step to, for folks that are moving to Kubernetes before maybe they go, uh, either vanilla Kubernetes or, uh, go into the public cloud. So, um, we are seeing a lot of on-prem Kubernetes as well.

Yeah. What do you see people doing that you just shake your head a little bit and say, folks, maybe we should be a little bit smarter than that. I mean, I'm, uh, I'm definitely biased here, but like, like people setting requests, like manually setting requests.

So our tagline at CubeCon, it's always on our booth and sometimes we're like, should we change this? But it still gets people, it still gets the conversation going. So it literally says stop setting requests.

And obviously I don't mean like, don't set your request, you should set your requests, but stop setting it manually. Um, and that is something I still, because most deployments are in the, like we have customers that have millions of containers that are spinning up and down every day. There's no way a human can do that manually.

So, um, and if you do, you're probably gonna then vanilla spread it across everything and send an average maybe a P 95. Um, so that is the the biggest thing I'll say of like, people should stop setting these things manually. You talked about predictive AI and using that to kind of manage the cluster, but will people also start using more gen AI tools themselves to manage the Kubernetes cluster and hopefully maybe something like an AI agent might explain how something actually works?

Yeah. Uh, it's a great question and something we're actively working on and, uh, I'll say arguing about as well. 'cause um, we, a lot of our customers have been like, yeah, I would love to use a chat bot, for example, to interact with the software.

Not, you know, I don't wanna teach my users how to go in and make changes or use annotations or anything like that. Um, but where, um, some folks are drawing the line is if it's a read action, so like, tell me about my infrastructure. Tell me the last time a recommendation was deployed, how much have I saved?

That sort of thing. Cool. But if it's a right action, they're like, I don't want the chat bot anywhere near that.

I don't want gen AI to then go make, uh, like real changes in my infrastructure. So it'll be interesting to see how it plays out for folks from us as a design principle, we've always said like, minimal amount of permissions, and then you can always add on if you choose as the user to opt in. So we'll still give people access to do things, but we won't have that as a default.

So then the, the folks, the, uh, it guys that are a little bit more conservative can kind of have control over how, uh, either agents or chat bots then interact with the software, Right? Because I don't think we fully solved this hallucination issue. So even if it's right, 99 times out of a hundred that one time it's wrong will be the most mission critical application in the stack.

Right? Exactly. And the most important thing in, in everything we do is trust.

Once you lose trust, whether it's ai, ml, it's just basic automation, once you lose that trust is really hard to gain back. And especially for the folks that are deploying software in their environment, they don't want their own internal users, the developers to lose trust in them. So no one wants to get paged in the middle of the night.

Um, and we don't want to cause that to happen. Do you think at some point we might have AI agents that are monitoring the activity of other AI agents and that's how we're gonna have layers of things to kinda, um, minimize any disruptions as much as possible? I mean, sadly, yes.

I do think we're headed towards a kind of crazy landscape. Um, we'll, we'll see where we actually end up, but I like, I think sadly that is a possibility. Mm-hmm.

Um, as we kinda look at this whole thing, is there an opportunity maybe now as Kubernetes goes mainstream to just say, um, can we simplify this in some ways at the, and maybe that's a job for the technical oversight committee, or is there another way to think about this where we just wrap Kubernetes in a bunch of things and there's a bunch of different projects because the TOC community doesn't seem to wanna get involved in that, the management of Kubernetes, if that makes sense. Yeah, It, it does make sense. I, I think we'll see a mix of vendors that try and solve the problem and the CNCF, uh, at the end of the day, like their resources are limited, so they're trying to do what they can with what they have.

And I do think the, uh, AI conformist document is a great first step, um, in this space. And we'll see in 2026 how much of that goes from, uh, like a self-assessment that is optional to more required things that should be in place. Um, but it's also opening ground for a vendor that has maybe, uh, some, you know, investment, uh, behind it to come in and provide that abstraction layer on top of core challenges that come up.

Mm-hmm. It also seems like maybe we're starting to see some separations of concerns here, because early on there was a Data Science Tiger team, and if they were lucky, they had somebody who knew something about infrastructure, and more often than not, they were unlucky. And the data science people ran into the same issues with Kubernetes, the developers did.

They said, this stuff's hard. I don't know how it works. And it's very complicated is the inference engines though, are now being separated from the training exercise.

And so, um, is that inference engine gonna be something managed more by a traditional IT DevOps team and that's gonna be part of the Kubernetes stack and that's how we'll go forward? Yeah, for sure. I mean, that's how we do it here.

Uh, I'll say formerly Storm Forge, um, of course the, the PhD machine learning guys were the ones that came up with the algorithm, um, and kind of set up both the training and inference engines. But now that we're kind of in a run IT mode, and especially for all of the data we look at, uh, each workload gets its own unique model. Um, and so we need various sorts of ML ops, uh, for our inference engine, and it's all run by the, uh, SREs.

Uh, sure the machine learning guys get involved sometimes, um, but a lot of that is actually managed, so all managed on Kubernetes and it is managed by the infrastructure team internally. And we've seen just, uh, for Storm Forge, we've been around doing like Kubernetes with machine learning for seven, eight years now. Um, and that skillset, uh, is, there's not a lot of, uh, ML guys that understand Kubernetes and vice versa, but we found it to be critical for being able to do what we need to do, and I think we'll see more of that.

Mm-hmm. So you've been at that for a little while now. What do you know now, uh, after having gone through all that process that you kind of wish you knew when you first started?

Yeah, so, uh, I will, you know, obviously credit for the team, they've been doing this for much longer than I have. Uh, I think I've been doing it for three, four years now here at Storm Forge. Um, but one thing that we all talk about of we wish we knew in the beginning is you can have the most advanced technology, the coolest machine learning algorithms.

None of it matters if the software isn't easy to use. Um, because at the end of the day, the people who are hands on keyboard have a lot of things they need to do in a day. Um, and learning how to administrate and manage your software is not top of their list.

Um, so I think the biggest learning for us, uh, luckily we, we learned that a couple years ago and, and got to, you know, put that in, uh, to the product. But it's something that is a design principle for us is that it doesn't matter how good the technology is, if it's not simple to use, it doesn't drop in within the ecosystem. Technology won't be used At the end of the day.

You guys are really at the forefront of automation in general and have been doing this for a while. What's next when it comes to IT automation? What can we expect?

Yeah. Uh, so what's really cool, uh, now, you know, being a part of Cloud Bolt is that the portfolio has expanded. So, uh, cloud Bolts, cloud management platform has been around for like 15 years now doing every type of infrastructure.

So being able to take our, uh, machine learning and infrastructure automation kind of technology and, and address that across everything is super cool. Um, and then vice versa, right? Like, uh, the CMP platform today does everything from zero to one.

So provisioning, patch management, um, ongoing day two operations and taking Kubernetes and putting that part of it in all the, the pieces that you need to do, like maybe you have to set up a, an OpenShift cluster, for example. Um, now we can kind of take a step back and look at the entire life cycle, which is really cool as part of Cloud Vault. Um, and then, uh, we have visibility into all the billing data.

'cause we have a finops product as well. So one thing that we will be sharing at, um, at CubeCon, and for anyone interested who wants to see it, they, they should totally come by the booth, but, uh, we are collecting the billing data for all the Kubernetes costs and then mapping that into your usage so you can see how much does this black box actually cost me? How do I split that by labels, by different teams?

How do I wanna share my, uh, all the shared costs, whether it's like idle costs, it's cube system, the things that people struggle with, um, being able to break down that visibility and then, um, go across your Kubernetes estate, uh, is kind of where we're headed and, and what we're working on right now. Um, if you were appointed the, um, Monarch of Kubernetes, what's that one thing that you would decide that we all need to do to change the platform or fix tomorrow that would make everybody's life easier? You have total control.

Whatcha gonna do Total control? I would, uh, get in place po resizing to ga. It's the, the thing that everybody asks for.

Um, not all applications, even if they should be able to take restarts, can take restarts. And, um, it, it's currently in beta, which is a little bit of a tease because it works in some scenarios, doesn't work at all. And, um, I think that'll be a big game changer for people.

All right, folks, you hearing it here? AI workloads are coming to Kubernetes. There's no doubt.

The only question is, is well just how challenging is that gonna be to manage and hopefully it's gonna get simpler. Yasmin, thanks for being on the show. Thank you for having me.

All right. And back to you guys in the studio.