Secure Governance and Scalable Management for AI Models with Yuval Fernbach and Adel El Hallak | swampUP 2025

Hey everyone, it's Alan Shimmel and we are live. That's right. Live, uh, it at Swamp Up.

Swamp Up is back in Napa. After I think two or three years it's been since they were in Napa s**t. And I'm thinking it might have been since before COVID that we were in Napa last.

But we're really thrilled to be here. It's beautiful here. It's a beautiful resort, but more importantly, there is so much going on at Swamp Up, you know, like everything else in the tech world, it's kind of the year of AI more than the year.

It's the era of a, the dawning of the era of ai. Still, I like to tell people we're still at the beginning of the beginning, not even the end of the beginning on ai. Let me introduce you to my first two guests of our Techstrong TV coverage here at Swamp.

Up to my far left. He's the guy in the, in the, in the, uh, shift happens. Frog shirt, Yuval.

Let me make sure I get it right. Excuse me. Yuval Fern back.

Yuval, welcome back. It's good to see you again. You good To see you as well.

You know what, before we get to our next guest, Yuval Give to share with the audience your title and role at jfr. Sure. So, hi everyone.

I am, uh, yba, I'm VP and CTO of MOFs here in Jfr. Um, actually joined Jfr, uh, a year ago is part of an acquisition of a company called Quack. Um, and nowadays, of course, part of JF Rog ml and the new product that we launched today that of course we'll talk about in a second.

Thank you, Yuval. To my immediate left not in the Frog shirt. Is is Al Elick.

You got that right? You got that? Perfect.

You got that on the money. All right. Ale is with, uh, Nvidia and Del introduce yourself.

Well, thank you for having me. Yeah, it's great to be at Napa. I was joking around earlier telling folks that, uh, you know, I'm glad we're doing this 'cause now my family really believe that I'm here for work proof.

So got the proof right. I got the proof now. So, uh, my name's Al.

I'm a senior, a director of product, uh, at Nvidia, and my job is to, um, take the software that our, uh, awesome core tech team creates, um, uh, harden those, make them production grade for enterprises and help our ecosystem build, um, agents, right, that are fra frankly transformative in everything that we do. Something we were just talking about. Absolutely.

And, and that's a great segue. I I bill something extra for giving us that segue. We were at the keynotes this morning, right?

You all led off Ale came on. You all, you, you, uh, introduced a new product for jfr called the jfr AI Catalog. Explain to our audience a little bit what it, what is it?

Yeah, so, um, as I shared, I joined JFO a year ago, and as part of that, I've seen and got a lot of responses from J four customers about the challenges they have with adopting ai, the challenges that they have with actually trusting AI and the amount of new models that are being launched daily, right? Um, everyone's speaking about ai, but actually using that in production require more than just, you know, testing the new and shiny model. It requires the ability to trust that, the ability to trust where that model is coming from, um, who's the owner of that model, and even who is actually going to use that model.

And as part of that, and as part of all that feedback that we received in the last year, we decided to launch the J four catalog. And that's basically a solution that allow organizations, allow our customers to manage the entire life cycle of AI usage, I'll call it, from discovering which models actually exist, um, to deciding who should have permissions to which models and eventually even serve those models. Uh, track the, uh, usage methods of the models and understand which application uses models.

And so the goal is eventually to allow organizations to understand where those models are being used by whom, and make sure that they trust those processes that are, you know, shifting in, in such a magnitude, in such a, a, a pace of innovation that we haven't seen before. Absolutely. We're gonna come back to that.

'cause I, I have some thoughts and questions, but not open. Explain to me the Nvidia Yeah. I mean connection, there's A reason for this awesome partnership, right?

Right. And so, uh, we're a full stack acceleration company. What that means is, right, uh, we're not just about producing processors or, or systems.

We actually build out AI factories, but we go all the, all the way up, right? For optimizing runtimes for not just models that Nvidia publishes, but also the ecosystem models as well. We call that nim nim inference microservices.

And so, uh, what we do, you can think of a nim as, as a, a model with a runtime package as a single microservice, we spend a lot of time tuning that runtime to make sure it runs it efficiently as perform as possible, uh, on the Nvidia stack. Um, but equally right, we contribute a lot to the open source domain. We're very, uh, we're huge participants in the open source community because going back to Val's point of having that, that trust, having that transparency, it isn't just that we provide the Nemo tron open weights, which are fantastic by the way, and Excel really good at reasoning.

But we, we also open source our, our training data sets. We open source our recipes so enterprise can then take those models further into them for their agenda, uh, capabilities. And so being the ones that provide the secure runtime and the open source of the models and the weights and partnering with Jfr, what drives the services for having all that lineage was just an amazing partnership.

Absolutely. I, I want to dive a little deeper on this, right? So I was at Swamp Up last year in Austin where they announced the, uh, J Fry Nvidia partnership.

Now Adel over the course of the 12 months, how have, you know, what, have you seen how this partner, well, look, AI has been on a hockey stick trajectory for these 12 months, right? But how has that affected, what's the, the, the net that our audience could take about this partnership? What does it mean to them?

I mean, look, you know, you've all kind of set the scene, right? There's so much happening and it's happening so fast. I joke around and tell people that at one point I think my kids thought I was a vet 'cause I was talking about new animals every week from llamas to Mambas to, you know, you name it, right?

But, but it's awesome innovation that's happening in the ecosystem, right? So a couple things that are, that I think critical number one is all this innovation that happens, right? Yes, you wanna be experimenting a lot, et cetera, but when you have all this innovation that's happening, right?

And you have all this open source, the potential for exploits grow significantly as well, right? And that's something we talked about earlier when, you know, we are on stage. And so, uh, being, having that transparency, understanding essentially what's part of your run times where malicious code can be potentially like implemented is, is super critical.

So you wanna be experimenting, but you also wanna be careful and prudent when you're experimenting. And so that's why having a single source of truth, right, for your system of records, for all your artifacts is critical. And that's why that relationship's been awesome.

And sorry, yeah. Go. No, no go.

And I think the second point is right, um, one of the first use cases we started using agent AI was in actually defining the contextual, um, analysis. Doing the contextual analysis to understand whether vulnerability can be exploited or not, right? And I think, uh, I really appreciate the partnership that we have with the JAR platform because that's something they take very seriously as well.

Just 'cause the CVE says, you know, it's got a high CVE score, doesn't mean it's exploitable. There's a lot that goes in to be able to exploit that. And so, you know, we see eye to eye in terms of how we go about really going deep and understanding the potential for exploits and protecting our, our, our customer base.

Absolutely. And by the way, this, this partnership didn't start because, you know, us and Vidia thought that we should work together. It started because the J four customers approached us, told us that they need to trust the source of the models.

And, you know, the only models for market face, by the way, I think the target phase is an amazing hub for models, but it's not enough in many cases. And customers approached us and told us that they want to have a trusted source of models, and NVIDIA is one of those trusted sources. So a year ago, we, we partnered to make sure that the J four customers can actually get the Nvidia e models, the directly for multifactor and trust the origin of those models.

Um, and from there, of course, we progressed with that partnership with the security solutions. So the contextual analysis, that ability to actually understand how those, uh, artifacts, how those models are vulnerable, and how we can make sure that in the production environment there will be no vulnerability. So eventually it's part of the same goal of making sure that the J four customers, and of course the NVIDIA customers can actually trust the models, trust the origin of the models, and trust that there are no security incident that will arise because of those new artifacts that they not need to manage and of course have to manage to actually make their product progress over time.

Excellent. Ada, I wanna come back to you 'cause you said something right in the beginning that I want our audience to understand. And that is, so a lot of people here Nvidia, and they're thinking G-P-U-G-P-U-G-P-U, not that you make a bad GPU, don't get me wrong, but the real key to NVIDIA's position is the software, is the community, is the ecosystem around Cuda and, and, you know, uh, um, NIMS and, and so forth.

Talk to us about that a little bit and why you are, we're on live tv Paul, uh, cameraman. I'm gonna ask you to grab outta my bag, my AI catalog paper. We'll bring it up.

We're gonna talk more about it. But al talk about yeah, what the secret sauce at Nvidia? Uh, Well, we're a full stack acceleration company, right?

I mean, um, yes, we, we start at, at the silicon, but we go all the way up the stack, right? And so we have AI factories, we have our, our software portfolio that goes all the way up to the, um, you know, what what we call blueprints, right? Reference workflows for how you'd go implement a specific use case for, for agents.

And you get the full benefits of Nvidia when you take the full stack, right? 'cause we're able to optimize all the way down to stack, but by no means you have to take the full stack, right? And we leave it up to our audience, our ecosystem to meet us where they think is best.

Some just wanna run on our infrastructure. We love them. Some want to utilize right?

Wanna go higher up in the stack and take advantage of the optimizations that we drive through software to enable that. I think one key to Nvidia is, um, you know, call it success or, or secret sauces, just how, how ingrained we are with the ecosystem, right? We, we go to market through our ecosystem.

Our partners such as J Fog are super critical to our success at the marketplace. And so you're spot on. We're not just a chip company, we're a full stack company.

Um, right. You can take us, you know, you can go with us up all the way, you know, all throughout. Or you could just choose to meet us where you think is best for your, for your, for your domain.

I love it. Thank you. So Yuval talking about the jfr AI catalog, you know, I've written a lot recently about what I call shadow I, uh, shadow ai, right?

And we've seen shadow, look, I've been in, I've been in the tech business probably longer than both of you. Okay. Um, I've seen shadow, before I saw shadow open source, there was a time where enterprise's official policy was no open source fill allowed.

It was a, it was a danger, right? I've seen shadow wifi. I remember being at a US Army base and the, uh, army Information Assurance officer telling me we don't have a wifi security problem 'cause we don't allow wifi.

And as I'm walking with them, I see people unplugging laps and throwing them under their desk. As soon as he walks by, they plug them back in. And wifi, we saw it with the cloud developers whipping out their credit cards and opening instances.

It's no different, no different with, it's probably even easier with ai. Yeah, you have pick your pick, right? Whatever one you want to use.

So we call this a prop, right? They gave this out at the, at the keynote today for your talk, your joint talk. Talk to us about the different models and how we're gonna control shadow AI at the enterprise level.

Yeah, yeah. So, so first of all, yes, it is a prop because, you know, this, this booklet have six models in it. Um, I believe that the current number in phase of models is around 2 million.

And you know, on top of that there are, um, external like model providers like OpenAI and others. So that's another couple hundreds of models. So, you know, the numbers are way more than that.

And of course, no book can actually, you know, manage and track the amount of models that are being launched. Um, and models are now they used for, you know, so many different tasks. So actually Shadow, um, a is in stock talk, talk about different type of models like reasoning models and, and, you know, voice models and models are being used for different tasks and not just for language models.

Like there are many models around computer vision and many models that are still used for structured data. And that's still a valid use case and still something that customers, you know, use as part of their use cases. Eventually the goal of the AI catalog is for organization to have the visibility about those models, about where those models are being used and how, and on top of that, as part of our launch, we actually launched a new product that will be available in a couple of months, um, called Shadow ai.

Now, the, the issue with shadow ai, the problem with shadow AI is that in many cases, you don't even know that the model is actually adding one of your packages. It's possible that you downloaded the third party doer image, uh, that dokey image that you use actually uses ai. Um, and it's not something that you can just, you know, not know about.

Because nowadays, even the regulators in many places, for example, in the EU, actually force you to show that part of your product uses ai. It's something that you need to have visibility on, it's something that you need to be transparent on. So the goal of the shadow AI product, but of course, is connected to the J four AI catalog.

It's to just not, not just allow you with Air catalog to choose which models should be used, but also to see, to have the visibility about where model is being used, what you are not really aware of. And if those models are being used, for example, malicious or those models are being used actually not approved in your organization, you'll be able to actually block those from being used or, you know, go through the pro through the process and approve those specific models. Um, so the goal is about visibility and The ability to discover where AI is Actually being used in your organization.

You know, again, my experience is you don't wanna stop people from using ai. Yeah. And quite frankly, stopping people from using AI is like trying to grab sand in your hand.

The, the tighter you make it, the more it slips out between your fingers. What you wanna do is just, okay, you're using ai, let's let us document it. Let's make sure it's safe, let's make sure it's secure.

Right? And that, because otherwise you're fighting a losing battle. Nvidia has to see that as well.

Al No, I, I, I mean, right. We're, we're not, we're not, we're not definitely fighting ai, right. To your point, right?

It's, it's, I mean, there's plenty of productivity gains new markets that it opens up, as I said, right? Uh, the, the only reason we're able to publish and maintain so many of these NIMS is because we're using Agentic ai. Right?

Absolutely. But to your point, you do have to be cautious, especially with all this open source that's happening, all this innovation, et cetera. You wanna create an environment that allows your developers to experiment.

That is for sure, right? You wanna, you wanna continue creating that, that experimentation, right? Uh, that you want to enable as well.

But then when you're going into, into production, yes, you want to have the safeguards that are in place. Um, do you want to be able to have the observability, the tooling that is in place, right? I, I go back to, you know, the, the nitron models that we provide, right?

Just being open source in terms of not just the model weights, but the data sets of what it was actually trained on, the recipes of how we got there. Just to give the enterprises and the ecosystem that level of comfort, right? To know exactly what's going on, right?

Such that you always have that lineage that's super critical. Yeah. I don't think you can, you know, on the contrary, right?

Like we're just on the, I think you called the era be The beginning of the beginning, right? And just imagine when physical AI comes into, comes into this world, right? Today we're talking about digital workforces, but very soon, right?

We're, we have these world foundation models where you're simulating and generating data to train these robots and these autonomous vehicles. Man, it's, it's about to get exciting. It it already is.

It already is. Um, but you know, I, that brings, both of you mentioned this, but you kind of ate at the edges. You didn't eat the middle, which is something else that they spoke about in the keynote saying, I'm trying to remember the exact name.

Was it AI gov or ai gov ops? Something? It Was, uh, dev gov Ops.

Dev gov. Ops, excuse me, dev gov. 'cause there's always something in the middle between dev and ops, whether it's psycho dev gov Ops.

I learned a couple new ones today. Yeah, sorry. Yeah.

Yeah. So, but that's really what we're talking about here. We want, we need governance.

Not, we're not here deporting AI models, right? We're here talking about you want use ai, use the ai, but let's have some governance around it. Let's have some guardrail, some knowledge, right?

And that's, to me, that's the enlightened way of doing this, right? We're not discouraging use ai. I know.

So textron's, part of Futur, we, we have a policy now where we're encouraging all of our people, the cameramen, the editors, the writers, the marketing, the decoders use AI to your hearts content experiment. We expect you to make some mistakes. That's okay.

Make the mis I'd rather you make mistakes. Trying something new than digging in your heels and saying, I I don't want to use ai. 'cause if you don't use, I tell young people this who ask me all the time, you're not gonna lose your job to ai.

You're gonna lose your job to someone who uses AI better than, that's Right. That's right. Alright, look, this is something we think about as well, right?

And kind of now you're, you're going above and beyond just serving a given model. You talking about managing the lifecycle of, of agents, if I may do that, right? Yep.

And, and that pipeline, right? We use, we, we have something called the NEMO platform for managing life cycles of ages. Mm-hmm.

And that starts from data curation, uh, creating additional data that is, um, doesn't have the potential PII that you know, you're not just collecting people's prompts, right? To then, uh, taking a model and adapting it for a specific domain. Then once you have that right, and, and putting it as part of a, of an agent, make sure you have the guardrails that are in place, right?

Such that it doesn't go re make sure you have the traceability. You can backtrack across the way. We have, uh, something called the NEMO Agent toolkit that allows us to drive all that traceability, all that observability, all that ping profiling around.

It's almost like, it's almost like onboarding a new employee. You have to teach them about your cultures and your norms at the company. You have to tell them the dos and don't dos, kind of like, you know, I'm doing in this interview.

Right? They, yeah. Right.

And so, so it's exactly like onboarding a new employee and putting all those kind of, you know, managing it throughout this life cycle. And to your point, it it first organizationally you have to, I love what you just said, right? Everybody's gotta be experimenting, but then making sure that your enterprise is leveraging the likes of the NEMO microservices to right.

To manage that entire lifecycle. I love it. Yuval, I'm gonna give you the last word and then we're gonna wrap up.

No. So actually going back to this, uh, dev gov ops term, and, and again, we talked about it today and, and this is in a way the theme of this swamp up because, you know, automation is already around. We're seeing that as part of the development lifecycle.

We're seeing that now as part of the DevSecOps lifecycle and also around, you know, ML and AI adoption. The challenge is not, or is becoming not how to automate those processes and how to actually, um, um, use new technology. It's how to make that in a governed way, right?

How to protect the way that we use the new technology and make sure that while we are researching new technologies, while we are actually adopting ourself to this new future, we do have the visibility and the governance on top of that to make sure that we're not doing anything wrong and that eventually our customers of, of our product can actually get benefit from those new technologies that we actually use in our products. I love it. Yuval del Thank you Del.

No, you got it. You got it. Alan, thank you so much for coming on here, kicking off our coverage of Swamp Up 2025.

We've got a lot more coming at you. Unfortunately, not all of it's live, but we're recording it all. And over the next days and weeks, you'll be able to see everyone we spoke to here.

I encourage you. com or Techstrong It Techstrong, AI digital CXO Cloud native, now even Security Boulevard. 'cause we'll probably do something about dev gov ops on there to, for our full coverage at Swamp Up.

But we're gonna take a break here. Stay tuned. We'll be back with more from Swamp Up This Tech Drunk TV.