Hyperscience CTO Brian Weiss on the Challenges of Operationalizing Generative AI

Hello, and welcome to the latest edition of the Techstrong AI Leadership Insight series. I'm your host, Mike, er. Today we're with Brian Weiss, who's CTO for Hyper Science, and we're talking about, well, getting ready for Gen AI because it's a little bit more challenging than we imagined.

Brian, welcome to show. Thanks, Mike. Really glad to be here.

We've seen everybody kinda launch one experiment after another, but I'm not quite sure that a lot of that is making it into production environments. And part of the issue seems to be is that it's not necessarily all about the technology, it's more about the rules, the regs and the cost and other factors that go into that. But what are you seeing?

Um, I see that a hundred percent and agree with not only sort of the, the stats, but the sort of trend that, you know, we start out with AI being kind of a, a, a solution looking for a problem. And while very, very promising for things like retrieval and summarization, the real rubber on the road now is, is, is, um, data inside the enterprise, right? That actually, you know, tells you about the language of the business or a process.

And of course, as soon as you do that, you, you're into privacy. You're into understanding like where that data's being trained, how it's being used, how you get access to it. So we see, I see blockers in twofold to success of AI projects and adoption is one, is is that sort of the, the, the concept of the unbridled use of AI to do every, everything in anything is, is, is, you know, needs to be kind of reigned in a little bit.

Uh, and then sort of the realization that the, the hard stuff is actually in implementing to get you to get to the data and answer questions about the data you care about. So I think the last stat I saw was, you know, that, that over, you know, 60% of projects that have kicked off to do something with gen AI have has stalled and they stalled for, you know, a lot of the reasons that, that you mention up front here. So we're seeing it in spades.

Um, you know, at hyper science we live inside the enterprise and we work extensively with really secure data. So things like veterans claims, things like, you know, information that, um, is very, very specific to individuals, uh, whether that's department of defense, uh, those kinds of things are, are, are mission. It's mission cri critical data where you can't be wrong, uh, when you're looking to get information out of a document set of that sort of thing.

So there's the criticality of the information and the need for getting it right, that I, I think a lot of the early stage gen AI use cases are, are, are banging up against, right? Mm-hmm. One of the issues that I think we're now confronting is the sins of our data management past.

And we all have structured data that, um, we manage reasonably well, but most of these AI models are being, or need to be fed something that looks more like unstructured or semi-structured. And well, if it was unstructured, we tended not to manage it all that well. So are we revisiting all of that stuff now and kind of, you know, dealing with an issue we, we probably should have been dealing with for the last decade?

Uh, yes. Uh, part of, you know, a lot of what we're encountering right now feels a lot like the early days of enterprise search, to be honest, right? I mean, enterprise search was, was not about structured data.

It was about, 'cause I can do a SQL query on a row in a column. The question is what does this thing say, alright? And how do I find the information in this 50 page document or a handwritten note?

So we have, I mean, TRA technology has traditionally struggled with all of that noisy information and it's kind of been a, you know, a north star that we've we're as you get more compute and now we have, you know, transformer models that can read things and do probability for what they understand and say and be able to respond. It's another chapter in that, but it is the unstructured data that it's, it's kind of the same problem, right? That if you haven't put some guardrails and structure around that for who can see it, how you can use it, what you need to do with it, um, then bringing the technology sort of full, you know, full bore to that is, is can be a real problem.

It's a struggle. Like I see a lot of the common struggles in gen AI use cases that, um, we're endemic to enterprise search. Who gets to see the data, right?

If I, if I load all this, this stuff up into my enterprise AI and does someone get to say, Hey, who makes the most money at this company, right? Um, but that, so document level security, all of the problems that are associated with who can see what and what's available and how it's available are all now trip wires in some of these processes. And while there doesn't seem to be a lot of regulations that's AI specific, uh, I hear folks will get down a path to a project and then suddenly they'll encounter something like HIPAA or whatever it is that they didn't think that they were gonna have to deal with.

And suddenly they're like, oh, wait, we can't do this 'cause it's gonna violate any one of 20 different regulations that are on the books. Um, how do we kind of navigate that so that we're not wasting time building things that are not gonna be used? Uh, I have a really strong opinion about that, and that is you need to work with AI and modeling technologies that you control.

So you control what goes into the model, what, how it gets used, and sort of the providence of that sovereign model. So we, we work extensively in government industries, financial services, where that, that, that's predicated on that. And in fact, it has been the blocker to being able to use some of the broader capabilities of AI now where at hyper science, what, what, you know, that's kind of a non-negotiable.

Like you have to be able to explain where you got the answer. You have to be able to understand the ground truth data that is being used to, to fine tune or train the model, um, and be able to really own the outcome, uh, around secure data. So, so my, my, I I think there's a, there's a, you're right, there's kind of a bifurcation happening here.

There are models that don't do that, right? Don't use them, right? Don't use them.

You use a, use a platform which allows you to select and tune and train and, and models which are accountable to not only the data they use, but also to the answers they give. Uh, and I I would say that they're, they're, you're sort of splitting two categories of models. Now, there are those for which I can do that and those for which should look if I'm gonna use them, then I, I'm, I, I can't get that accountability or, or, uh, transparency.

So we, we have been building models for many years, uh, for in, in-house, in some cases air gapped environments, right? That look at, you know, say for example, uh, healthcare claims at the, at the Veterans Administration, right? These are complex boxes of documents that have handwriting and all kinds of stuff all over them.

There's no ter external modeling to usable there, right? We need to be able to be on, on site at the va. And, uh, I mean, we're, we're, we've got models now that, that deliver, you know, AI results at 99% accuracy.

And we've taken the, you know, the processing time from months down to days. Uh, but all of that's contained. Like, like you can explain not only the answer, but also how it was trained and, and the way it's being used in combination with those techniques.

So I, I see it, I see the market maturing and, but you're, you're absolutely right, Mike. There's a, there's lots of 'em that stall where people get excited about using a frontier model. And then, um, look, the new, the new InfoSec gauntlet is, is your AI review committee.

What model are you using and why? And what is it doing and who owns it? And where's my data going?

Like this is a, this is now the new normal, right? To have to really vet any kind of model inside an enterprise extensively. The other issue, or at least one other issue that I keep hearing about too, is people will get through the pilot and then they'll go into production and they will have grossly underestimated the cost of running the thing.

Yeah, Yeah, yeah, yeah. I look, that's another market maturity thing, right? So if you think about it, the hyperscalers who are trying to, there's a sort of this big land grab to become the model that everybody loves and it's being underwritten, right?

And as soon as you have to think about using that at scale, there is an underlying cost that's actually very, very hard to accommodate, right? So people get excited, like, I'm gonna use this giant model to do this task that used to, you know, but why would you use a helicopter to cross the street? Like, what?

We're like, I'll use that to cross the canyon, right? But if I, it's not, somebody's gotta pay for it at some certain point. So you absolutely see these things like, wow, it worked really great, and then you realize that you actually scoped in a way which is just financially unreasonable.

Um, so I, I see that all the time. And, and you know what, what, what I'm focused on as, as sort of the, the composable platform at hyper science is using the right tool for the jobs. So, you know, let's use the CPU driven trainable models that are, understand your data and get you a really great result for the price.

And then I can then stack lots of things that are way more complicated, read, more expensive, GPU driven, all that kind of thing. But, but now I'm gonna start to decide like I want the best outcome for the right price using the models that are the most effective. But I see that all the time, super excited.

Let's use a, let's use this giant model to do this thing. And you realize like, oh my God, I just took a helicopter across the street. I can't pay for it.

Like, why did I do that? You know, all the time. Do you also think that maybe, you know, to your earlier comment, will AI push more people to something that feels like a private data center, whether it's on premise or a private cloud or something?

And, um, we're all gonna be seeing a lot more of that activity rather than just relying on a public cloud. Uh, that's already happened and already happening, like this first wave of AI workloads. Most of our clients, most, 'cause we're dealing with government entities.

You're dealing with anybody who's, who's rightfully concerned about the providence of the data and the ai the workloads are going on-prem, right? They're going, these workloads are going, so you, you see the industry responding to that problem. But these init, this initial set of workloads on private done data that needs to be managed, uh, carefully, um, is, is on-prem.

Like, it's, it's, it is shifting, you know, that that sort of grand move that we all had to the cloud, like everything's gonna go from managed to just 100% public cloud and I'm gonna buy it by the minute and consume it. And, but now all of a sudden it's a, it's a very, very different, uh, um, process. And, you know, we're in kind of a unique spot at hyper science.

We, we deliver OnPrem, we driven private cloud, we have a, a FedRAMP high secure SaaS environment. Uh, we are e we are partnered with some, some, you know, the hyperscalers in, in, for example, Google's effort to provide a managed on-prem service of their, of their models, right? Is also embedded with hyper science.

But yeah, I, I think it's kind of the old is new again in that regard. And, and it's, it's a hundred percent understandable. Uh, Is, is this therefore gonna become something of a rich company's game because you're gonna need to buy the infrastructure set up those data centers, get all that data managed.

I mean, none of this stuff is inexpensive. So, um, you know, what can a smaller company expect to be able to do versus a larger enterprise that has the resources to drive this thing? Yeah, I'll go back to my helicopters across the street.

You don't actually need, you can achieve really high results, uh, and high performing results with narrower models on an ensemble which are actually cost effective for the task. So you don't necessarily have to, you know, only the, only the, the really rich people can afford the, the machine, which, which will get you the, the right and the perfect answer. Like that's actually, it's going the other way.

What we're starting to see is that a composable architecture where, um, a combination of models that are cost effective, and then you bring in the ones that are more expensive to, to do workloads that make sense. You know, I've got a 300 page, uh, credit swap agreement with nesta tables and handwriting all over it, and there are gonna be chunks of that that are really relevant for a a, you know, a a large model. They're, at the end of the day, they're, they're, you know, they're, they're probability calculators for language and they're big ones, right?

But I don't need, I don't need it to tell me the square root of 32. Like, I don't, I need a, I need a calculator from, from CVS to do that work. So don't ask that.

Don't spend your money there. But I, so I, I don't, and then, you know, the other point there, Mike, is that, um, look, the, the, the amount of innovation driving costs down, I mean, that's, it's, markets do that, and then technology market does it, it's the same thing we saw with the virtualization of CPUs and, and, you know, when cloud came out, like, you don't, so that, that is happening and you'll see the, the, you know, the compute get stronger and the price get driven down because there's so much pressure to make that possible. So over time, I, I don't, I don't think we end up being in a, in a, in a class warfare situation here with it, right?

The market will respond. And if people are smart about what you know about, about combining the right tools and not trying to get, you know, you know, buy helicopter to go everywhere, uh, I think gonna be all right. Will there be a shift in demand then, based on what you're saying?

Because right now when I encounter people, you know, all they want is the latest and greatest GPU and they forget about the other GPU cycles, but before that, and those ones are a lot less expensive, and there's also other classes of processors. So are we gonna get smarter about all this stuff? Uh, yes, we are.

And I actually think that the, um, the market will create that for us. Um, so you're already starting to see companies that have sort of have bet the farm on, on being able to underwrite the GPU, but a fixed cost, but yet give the, if you can, if I'm, if I, if you get unlimited use of A GPU at a fixed cost, what is my business model if I can't charge you by the minute? And you can use it as much as you want.

There's a, there's a sort of a, a vicious cycle that I have to get ahead of, so that, that trend to sort of, everybody wants the latest and greatest, and there are some artificial pricing things happening right now with, you know, fixed cost against unlimited use and things like that, that are, and you know, we're seeing people go outta business, um, as a result of that. So I think what happens is that, that everything just kind of, that you can't keep that, that you can't stay ahead of that curve. It's almost a Ponzi scheme, right?

So what will happen is we'll end up with, um, we'll, we will end up with like, well, yeah, the good, the good enough model is, is good enough because I can afford it, right? As soon as the really good ones I can't afford anymore, and we kind of artificially are being able to think we can afford them. And then once all of that shifts, I think people will quickly say, yeah, yeah, I, I I need the right answer for the right price.

Not, not I need to use the greatest thing on the planet to get the same answer. Uh, Um, we also, you know, once again, seem to be thinking about security as an afterthought here. And a lot of the deployments that I've seen so far have, you know, significant vulnerabilities and there's all kinds of new ways to hack into these AI models.

Are we waiting on some sort of catastrophic event before we get serious about AI security? Uh, I hope not, but maybe I hope not, but maybe, I mean, it's the, um, and I think, uh, uh, you can go back to other sort of inflection points in, in technology that have been somewhat similar, the internet, right? Things like that.

And, and each time that happens, you create this surface area, uh, attack surface for, um, for bad actors. And I, I do think we are, it is running very, very fast. Mm-hmm.

You're not wrong. It's running fast. And I do, I do agree, and I I I am concerned there's risk in that.

Do I think that, um, you know, it'll, it'll take a major event to, uh, to snap everybody in the line. I really hope not. Uh, the trend I see in large enterprises is they are getting very serious with, uh, AI governance boards and security committees to try and keep up with it, uh, in the security industry.

And it's a, it's a whole new world of, of vulnerability. So I see enterprises reacting and trying to be sure of that and get ahead of it. It's also, you know, there's so much new every day that is risk.

Mm-hmm. Also, early on, at least it seemed to me, most of these AI projects were led by so-called Tiger teams, and they pulled everybody together and they even had dedicated IT people who knew about infrastructure. But is more and more of this AI workload gonna just be shifted over and managed by traditional IT teams?

Or will we always need tiger teams? Yeah, you're seeing the impact of innovation, right? On business structures.

Uh, and it, i it will level out. I mean, you don't, you don't have to be an AI expert in order to be able to look at hyper science. We, our platform is developed for just ordinary business users to be able to train models, um, and get high performing results.

So the data sciencey part of this is, is being democratized and productized, number one. Um, and then as we find those use cases that really matter to a company, I mean, there's a lot of stall like, you know, AI projects that are about, you know, building agents that'll do magical things someday, if you can figure out how to justify the ROI. And then there are are folks, and we're in this category where you're actually just creating hard ROI in the business.

And so as, as businesses find the use cases that, that deliver, you'll, you will, you will see it codify into a sort of run rate IT function. I don't think the tiger team who specializes in all things AI is a, um, is a new and permanent sector of the enterprise. Hmm.

So having considered all these things, what's your best advice to folks? What should they be thinking about right now to kind of avoid what are se what are some serious pitfalls? Um, look, the first one I say is, is know your ROI target.

When you go in, um, there are a lot of phishing expo just like, you know, science experiments and things like that, that end up being an, and we all know the downfall of that is that, you know, you're, you can't justify what you've done or how you've done it. So there are, there are drill sites for real value with AI driven, uh, um, you know, opportunities. And, uh, I would start there.

Let's, first thing is let's just sort of understand the outcome and justify the business outcome and know how you're gonna get there. So I, I'd say that's the first one. Um, I would say, uh, if anyone tells you that one, one model or you know, one vendor's got the magic thing that's gonna do everything, then you probably just like think twice about that, right?

That the, you know, using models, plural, using AI in a com, in a, you know, composed way is gonna get you better results and also more control. Uh, so I'd I'd say look at a composable approach, um, and then the last one, I think you called it already, which is data security and, and the providence of what that model is working with, what you're using it for, uh, and, and ensuring, particularly if like, we're, we're in, we're in the, we're in the really, really high fidelity mission critical data business, right? So, um, it's one thing to ask a, you know, a model to go do research for me and summarize things and all that kind of stuff.

But if I'm asking it to, you know, process information, uh, and make decisions potentially that are mission critical on data info, you can't really be wrong. Um, so you gotta think about the right, the right tool for the job. What is that, what is that model supposed to be doing?

And, and I would say above all, you know, managing security and privacy, et cetera, is that model accountable for when it's wrong, right? So when it's wrong, do I know, can I solve for it? Will the model in the process or what the platform itself help me solve for that?

Um, oh look, maybe you don't care. Maybe maybe error rates are great, right? It doesn't matter.

So you get the sentence a little bit different. And what if you're just sort of generating content and reading books and doing summarization? It's something.

But when you're into mission critical data process, you really sort of look at the whole picture of, of what, what do you do with wrong? Or what does that pro that mono, that process tell you about wrong? And does it give you the tools to bring people, say, for example, to solve it and sit next to the model?

I think that because the frontier of this is not models do everything, or AI does everything, it's a combination of, of governed models working with the right slice of the, of human intervention to ensure you have not only data quality, but security and governance and, and transparency to the outcome. So I, I gave you a lot there, Mike does that, I'm probably gonna have to go to ask GPT to summarize all that for me into the main points, but You know, I, I think you heard it here folks. I think based on the goals, the risks and the cost, you gotta make sure the AI price is right.

Hey, Brian. Yeah, thanks be of the show. I guess that makes me a human, GPT too.

There you go. Thanks everybody. Thanks for watching the episode.

You can find this episode and others on our website. We invite you to check them all out. Until then, we'll see you next time.