Nutanix Enterprise AI Demonstration

Hello everyone. Good afternoon. Uh, so my name is Laura Jordanna.

I am a technical marketing, uh, director of technical marketing at Nutanix. Um, and so for the next few minutes I'll be walking through the demo and Ashwini and Jesse will be here to, uh, help support as well. If there's any questions, I'm gonna sit down, if that's okay, because it's kind of hard to bend over a little table here, Laura Al.

Yeah. Um, but as we can see here, right, this is, uh, the interface for Nutanix Enterprise ai. Uh, so it's a Kubernetes application.

Uh, it can essentially run on any Kubernetes, uh, so it can run on our own platform and, uh, Nutanix Kubernetes platform. Um, it can also run on EKS or, uh, other Cloud Kubernetes instances. Um, but here, so what we have here is I'm logged in as the admin.

So if we were to kind of take a, a day in the life, uh, a user story of, uh, I'm an admin, an IT admin say, and, uh, I've been getting a request from one of my developers I want to deploy. They want to be able to develop against a specific model or, or, um, they want an endpoint that they can test against. So as the IT admin, I don't have to understand all the capabilities and, and understand data science and what they're actually doing on the backend.

I just ha need to know how to log into the Nutanix Enterprise AI interface, download a model, serve up an endpoint, hand that over to my developer. So I'll walk through kinda how easy that is. Um, so looking here at the dashboard, logging in, right, as Swinney was mentioning, we can see, uh, all of the endpoints that are active, right?

What are the top five talkers? So I can see like who's getting the most requests. I actually do have deep C here, even though Sweeney said we can't use it.

But again, we can import our own custom models. Um, we can see which API keys are being used the most and what the trends are looking like in terms of, uh, successful and valid requests, failed requests, what the health of our infrastructure is. So that's really important for an IT admin to understand, um, the impact of these endpoints and models, uh, on their infrastructure so they can kind of monitor the usage, the CP usage or all of my nodes here.

Um, can I ask a, just a quick question? Sure. What do you consider an endpoint?

So the endpoint is, is like a, so if you're using like chatt PT or something like that, right? And you're, you're asking a question, it's connecting over an API mm-hmm. To, to, uh, that, that the model's being backed by that API.

So, uh, if we walk through, it'll make me make a little more sense. Um, okay, So if I'm connecting, you know, making a query to when the models am I an endpoint, So you're connecting to an endpoint. So, yeah.

Okay. Yeah. So, um, so coming over here to the, the models page here, um, so we have right now a bunch of models that we've imported or connected to already.

So one of the key benefits is that we can connect to, um, you know, multiple types of model hubs. So right hugging face, right? We've, we've validated, as Swinney mentioned, um, several models here.

Um, I can't unlock the brightness, but that's fine. Um, so we have a bunch of models here. So for example, I can select the, you know, a small model.

I can click on import, I can put in the name, I wanna give it, hopefully that's not already taken. And it basically, it'll start, uh, it'll carve out the storage, uh, on the backend, download that model and get it ready to be, uh, used in an endpoint. Um, so what that looks like is, and, and again, we can also, uh, imports custom models manually if, if it's not in that list.

Um, on the Nvidia NIM side, right? We validated a bunch of models here. So these are models that can help support agentic workflows, right?

Like, um, embedding models, uh, safety models, re-ranking, et cetera. Um, and we're continually adding to that kind of, that validated list. Um, so once you have a model ready, so we have, you know, a bunch of models that are already active, we come over to end points, and this is where we actually, um, create that, uh, inference engine connection to that model.

So the model is just a bunch of files, right? We need to create the access to that model. So we come here to create endpoint, right?

We give it a name, um, usually a better name would be good to kind of match what we're actually doing here. But, um, so we can, you know, for example, look at the, uh, mytral model here. We can select what GPUs we wanna use, um, the number of GPUs per instance, right?

So if we have a large model, um, we can, we can split that up. Um, and one of the key benefits, as Ashwini mentioned, is we already fill in kind of the configuration so the admin doesn't have to figure out what do they need to be able to successfully run this model. We've already kind of done that heavy lifting for them.

A question, just, excuse me, just back on the endpoint name. Could I think of the endpoint name as just maybe the, uh, the application that I'm going to be serving or the other part of my infrastructure or application that needs access to this model, right. That I'm gonna be submitting information to?

Yeah, I mean, I think it, it would make the most sense to kind of name it after the model that you're using. Um, and then maybe the API key would be tied to whatever application's using that model. Right?

Gotcha. Um, so yeah, so let's rename that to menstrual seven B, so it kind of aligns more with the instance name. Um, we, we give all that information and then we add the API key.

So we have some API keys already created, or we can create a new one here. Um, so I'll go ahead and just create a demo key, copy that out. And then, um, once this model gets, uh, once, once this endpoint gets deployed, it will be in a pending state, uh, once it gets deployed, then we'll be able to use that API key, um, and then use the information to connect and use that in our application.

Um, so to kind of, to speed things up a little here, it'll take about five minutes or so, um, we'll use one of the models we already have. Um, so for example, we have, again, deep seek and this llama 3 8 1 beam model that's already in an active state. So as the IT admin, when I see that it's active, I can can come in and say, okay, the model's active, let's test it out and make sure it's good to go for our, uh, for our developer so we can test it and see, okay, it's returning a response, it's not giving any errors.

Um, so we can be kind of satisfied that it's working, right, so that they, they know that it's working. And then they can come over here, gather the API key, um, view the sample request, and hand that off to their developer, and then use that in your application. Um, and so these models, right, that support, uh, those EnTec workflows, right?

Um, we can, you know, include the tool calling in the, in the sample request as well. So it does have that capability. And do you wanna wanna elaborate on that a little bit or?

Sure, Sure. So, um, uh, I think this comes back to the whole agent story where, um, there's, there's a lot of buzz in terms of everybody writing a tool that can, you know, be invoked by ai. And so this is essentially enabling that tool calling, um, where the request actually registers the format in which the tool is gonna be exposed, and then the LLM is able to make a call or make a decision on which tool to use.

Can, uh, hi Jack, Paula Paradigm Technica. Can you, uh, is the, uh, API key only embedded in your ui or can you do stuff with key management and share just sequence managers So you can, I mean, you can, um, of course copy it out to a key manager. It's not have integration today, but, uh, that's Seems a little risky, so something No, no, you're absolutely right.

Um, and, and I think the concern we had was, I think what we heard was most customers already have a mechanism baked in for keys. So, um, it's a little bit hard to get them to kind of migrate over to a new, uh, way. But you, you're absolutely, yeah, that's a great to integrate, yes, To integrate is important, but yes, to have the key embedded in the product and not in a protected vault that oh, is just wait, You have to, it's not embedded, It's not in plain text or anything like that.

Right? Sorry, sorry, maybe we didn't get your question. Well, It's, it seems like from the user interface that the API key is right there, so, well, it's, It's, it's grayed out, right?

Yeah. So once it's been generated only a few and you've copied it, much like other platforms, You won't be able to, it, you won't be able to access see It again. So the only way to access it okay, yeah.

Is Yeah. 'cause It's just a bunch Of, you would have to rotate it if you want to get a new key if you lost it. Yeah.

So it isn't accessible once you've generated it. Okay. You can only copy it the first time.

So it's not Just masks there No. Just saved the parts there that we're seeing with the mask on it. Correct.

Right. Yes. You're, it's the key isn't saved and retrievable.

You're right. And, and one thing you can also do, if you believe A API key has been compromised and rotating is not an option, delete, you can actually deactivate it. Delete, yep, yep.

Sorry. Yeah. So all the API keys are managed here.

Right. And you can, this one I think I just created, I can just deactivate it since I'm not using it anyways. So at this point, when you turn this over to the AI developer, I guess that's the term I will use there, um, they can write an app against that LLM.

Yes. Yep. What about all the other capabilities that we talk about within the AI data analysis, whether or not we're going to add in more augmented data into there, whether or not we need to train it on our data.

So I've got an LLM here, but I've got some private data that I wanna train it on and tune it up with, et cetera. How do I do that with your system? Got it.

Um, so in terms of training, so this one is actually strictly an inferencing product. Okay. But then we do allow training on the Nutanix platform.

So, um, the Nutanix platform, of course has storage, which is high speed, and we actually have ML per, uh, benchmarks that we kind of run. Mm-hmm. And, um, so we have a platform on which it, it's training ready.

But I think, um, the, this product itself is just for inferencing, basically because, um, I think most of our customers, they are looking at consuming LLMs rather than training them, uh, from scratch. And does this product have a name other than Nutanix? Nutanix Enterprise Ai.

Enterprise ai, yeah. Okay. That, that just rings straight there.

Okay. It should be enterprise AI for inferencing, because if you have the other ones then, because you would think if it's enterprise ai, you were thinking that it's covering the gamut, but you're not covering the gamut with this. We, we could in the future.

So the thing is, we're new, we're the CLE on the block and we're looking for feedback to evolve. Definitely Discuss, I think as, as you guys continue to build out the product. Yes.

You know, if you incorporate, let's say rag components, then it would lock us in like we wanna have Room. Yeah. Then we don't have a way out.

It comes up, We A marketing nightmare for the folks in the back. Yes. Sorry, one Interesting question, Jim Rinky from ZDCI was looking at the list of models.

You had the initial list of 17 models, and some of those are more failed. I'm curious what that means. Is it that if it failed, which is fine.

I mean, is there, is there Yeah. Mean, There there's little tool tips next to it that tells you why. So in this case, our, Yeah.

Oh, so like you just failed that one token, right? Yeah. So whatever.

So now anything that This is, this is actually a different token. Different Token. Oh, okay.

Token to pull them out. Oh, that's A, That's a good point. So we, one thing I didn't show, oh yeah.

One thing I didn't show is it's fine. This is where the credentials are set for. 'cause you need a credentials to access these repeats, right?

Sure. So you set those credentials and apparently justice is wrong. 'cause and so you put 'em here And I save my keys all the time.

You, I could see having just done some of this in another cloud, I could see where, you know this, why isn't it working for a, a data scientist, right? It's just like, I just need to get in there and start use. Oh, don't even waste your time with that.

Yeah, yeah. Right, right. Is is excellent.

Exactly. Yeah. Excellent.

And some additional error handling would include, like if you haven't accepted the end user licensing agreement. Yes. So for particular models where you need that, it actually tell you your hugging face token is valid, but you haven't accepted, you don't have, Okay.

Yeah. Right. So at this point, this endpoint, I could originally, I could start throwing a, a query against this and see how this thing works.

Yep. Yes. Yep, Yep.

Yeah, just with the, with the, um, if we come back here, yeah, we did the test, uh, and then, you know, we give a sample curl command, uh, also, so you can, you know, take this plug in your API key, and then, And at this point, am I going out to the public? No internet at this point? No.

Still inside my firewall. Yes. Yep.

So this, so this is completely inside your data center? Just inside my data center, yes. Okay.

Yeah. Excellent. And, and the good thing about this API format is it's OpenAI compatible.

So even though you're not going to OpenAI or the cloud, your endpoint within your data center is now speaking the same API. So if you have an open source project that is written for OpenAI, it'll just work and you can consume it. Yes.

So there's a lot of like tools. Sorry. No, go ahead.

There's a lot of tools, like anything LLM for example, right? You can, you can connect to any endpoint and there's like an open AI custom or generic that you just plug this in and it works right away. There's no Tweaking to ano.

Another advantage here, of course, is that I don't want anybody sending out anything proprietary, right? Yes. Right.

Exactly. By accident to open AI and reveal the formula to my Yeah. Widget, radioactive code.

Keep it all, keep it all local. Yep, that's right. Yes.

When you're pulling models, is there one token for hugging face or can I have separate tokens for different models? So It is or Is it always his It's it's pro, It's per user. It's per user, yeah.

So you can have, one thing I didn't show was you can have, you can create multiple users on the system, um, that are either also admins to that, that can see a global view of everything. Or for example, if you wanna give just access to like a data scientist or developer, so they can just build their own or create their own endpoints, you can do that as well. And then they have limited view, but they use their own tokens and things like that.

So I'd like to ask a couple questions on the infrastructure side of this solution. Um, can you elaborate, does these runs on top of a HV or VMware, or does these runs alongside a CDF on Nutanix? Do you need to have specific kind of Nutanix nodes to run that?

And also, and maybe it's a, a bit more of an extra question to that, or is that kind of license you need to have an extra subscription to use that? Yeah, so I can answer the first question. So it, it's a Kubernetes application, so it can run on any Kubernetes, uh, basically CNCF certified.

So, uh, for Nutanix, that's gonna be Nutanix Kubernetes platform. Um, we can run on Amazon, EKS or, or Azure or um, uh, Google's, GKE, um, OpenShift and other ones as well. Um, and, and so it, it runs on top of that.

So, and that, again, if it's NKP, it's running on, uh, Nutanix, a HV, but, um, yeah, the cvms a little bit, so yeah, But in this, in this particular environment, it's, it's running 100% on top of an EKS cluster. Yeah. Yeah.

So, so to answer your question, it's actually running on zero Nutanix right now. Nutanix Infrastructure, if I, Okay, Good. I apologize.

Hardware you, you're still running Onan Nutanix software. Okay, so hang on. Now, now, now you've lost me.

Okay, so physically, what am I looking at if I'm looking at this platform? Yeah, maybe can we repeat that thing? Just the laptop that's connected to the cloud.

Oh, yeah. Just to, Yeah. So it's running on this Kubernetes cluster right Now on a, so it's running on servers, it's running on servers somewhere.

Yeah, we, we have an on-premises as well as cloud. Okay. So basically we wanna give people anywhere you can put a Kubernetes cluster.

That's The, that's the point we're trying to get at. So anywhere you have a Kubernetes distribution deployed, okay. The idea is that this gets deployed as a helm chart much.

This obviously says EC2, but this is an EKS cluster. So each node is an EC2 instance under the covers. And in this case, if you look on the far right, we have a total number of GPUs of six.

This, this guy right here is an L 40 S node pool with four GPUs per node. So it's actually only a single node with 48 VCPUs, 372 gigs of memory and four GPUs on a single node. And then these two guys are L four nodes each.

So this, this, uh, these two nodes have one GPU each, and this guy has four GPUs on a single node, which basically allows you to leverage tensor parallelization and deploy larger models that require more than the one or two GPUs that they may need to, uh, run their instance. Um, the, the other thing to, to kind of touch on is when we download our models, it's leveraging CSI drivers storage classes to, to effectively deploy the models. And in our case, because these models can be deployed with replicas to enable ha, it, it, we typically, uh, require, uh, files based storage.

So like NFS rewrite many, so that multiple models can basically run directly from that NFS export and, and read and write from that one endpoint, from that one NFS share. So We will see that, uh, all is based on Nikki S cluster, right? Yes.

Or, uh, a Kubernetes cluster Yes. And body integration, are these, uh, exposing API or is there a CRD or what For it? Under the covers there's CDs, there's controllers.

Yeah. So for example, the, you know, the downloading of the model is leveraging a model controller, which basically takes care of making sure that a persistent volume claim is created. Okay.

Leveraging the storage class you define. So in our case, because we're running within AWS, we're leveraging EFS. Yeah.

For NFS, you know, rewrite many, uh, use case. And then subsequently, um, if you're in a Nutanix environment, you'll be leveraging Nutanix files and so forth, respectively with other, you know, providers. Okay, gotcha.

So the whole idea is if you, if you need storage, you can leverage storage from Nutanix or you can leverage the native storage that's given provider From a platform engineer perspective, from platform management perspective in general. So this thing should be integrated with a, you know, master dashboard or the implementation that the customer already has. So for like moti fleet management?

Yeah, yeah, Yeah. So, so one, one of the, you know, uh, uh, solutions that we provide with NKP is moti, uh, cluster fleet management. So that's actually, you know, one of the kind of, uh, uh, driving, uh, scenarios for customers who are looking to not run in a public cloud, but are looking for an enterprise class Kubernetes distribution.

Yeah. That's where we typically would, you know, approach them with D two iq. And I'm sorry, with, with, with the D two acquisition, uh, we, we, uh, are now, uh, providing NKP as a platform and that gives you a lot of the, you know, multi cluster fleet management, uh, GI ops workflows, the CICD workflows, if you will, the, uh, integration.

And obviously there's tighter integration with Nutanix platforms, so there's things that you could leverage within Nutanix that would become a lot easier to consume. So when you're thinking about, and, and earlier we had asked a question around like, like, you know, Nutanix ai, that's, that's kind of the name, but some folks, um, the previous versions were, were referred to as GPT in a box, because the first idea was that we provide an appliance that had everything that you needed from, you know, virtualization. So the hypervisor from a storage perspective, from from Kubernetes perspective.

So in one appliance, you can effectively deploy everything that you needed to, to provide a fully air gap disconnected environment. Gotcha. Thank you.