Challenges of Operationalizing GenAI with Christian Lau

Hello, and welcome to the latest edition of the Techstrong AI series. I'm your host, Mike Bazar today with Christian Lau, who is Chief Product Officer for Dynamo ai. And we're talking about, well, now that we've all kind of gotten our heads wrapped around ai, the hard work really begins.

We gotta deploy this and manage it and govern it and secure it. Christian, welcome to the show. Great to be here.

Thanks, Mike. What are you hearing from folks in terms of what their challenges are? I feel we went from couple of, maybe 18 months ago, a lot of irrational exuberance to a lot of experimentation.

And now I think folks are kind of trying to figure out, well, how do we, uh, deploy AI into a production system? Yeah, I think when you look back 18 months ago when this new technology was coming out, uh, there were a lot of questions about what is the, what are the risks that we even need to account for? When you have a chat bot that can basically act like a human being, you can prompt it in infinitely different ways, and it can output, you know, many different things.

It's, it's, uh, what we call like an unbounded space of a possibility that you can prompt, you know, and so, uh, to a lot of our customers that we work with that are in more regulated sectors, like say the financial services sector or the federal or DOD sector, you have a lot of questions about how you apply existing, for example, model risk management to this new technology. And I would say over the past 18 months, you saw a lot of progress here because in order to actually deploy this into production, not only do you have to get the technology to work, but you also have to prove to many stakeholders across the organization that this is a technology that's gonna be secure and compliant when you actually deploy it, right? And it's gonna, uh, you know, not harm your end users or harm your company's reputation.

And so that's kind of been a, a big focus that, that we've, uh, assisted many enterprises with, is how can you generate evidence that you've properly managed the risk associated with the technology? And that's really the first step to, to unlocking the true value of all the amazing work that's being done at the Found Foundation model LA layer and application layer we're seeing today. Um, that that's kind of like, uh, what, what we're commonly seeing across our customer base.

How do I govern all that? And I'm asking the question because AI models are subject to drift. And so I may finally build one and deploy one, but it may not be acting as expected, you know, nine months after I've deployed it.

And as far as I can tell, it is not like traditional software where I just apply a patch and update it. I've gotta kinda, I, for lack of a better phrase, rip and replace. But, um, how do I kind of wrap my head around that workflow?

That, that's totally right and kind of dive a little bit deeper into that problem. There's different aspects to this, right? So when you look at the underlying what's underneath the hood of most of these AI applications, you might find that there's a rag vector database attached to it.

Uh, you might find that the large language model might actually be swapped out at different points of time, might be updated by your foundation model provider, um, or it might be dynamically switching. And so all these things really affect the drift of these models, and there are ultimate performance, right? And so it's really important that as time goes by, you implement two sets of controls that we recommend to, to, uh, enterprises we work with.

One is doing repeated and very custom and tailored evaluations of your AI systems, of your models, uh, even behind the scenes, right? Uh, so this means that running, you wanna run a repeated test when the newest model comes out, are you gonna be able to run an evaluation, not just the general benchmarks, uh, but also benchmarks are gonna be tied to the specific use cases that you care about. Because a lot of times you'll, you'll find that the latest and greatest model that comes out may be really great at coding, but it might actually be worse at, for example, writing the types of reports that your particular use case calls for.

So you want to build these customized evaluations that kind of run in an automated way and in a, and a repeated way. And, and we'll give you consistent, we'll check if your, your model is giving you consistent results, right? So that's on kind of one side, these types of offline evaluations.

But the second part that you wanna implement is real time monitoring and observability, right? And this is where we really start to see now these AI systems, these, uh, you know, these models actually become products in themselves. And you want to apply, you know, as I'm, I'm I, I lead the product team at Dynamo, you wanna apply the same principles of product management, of, of gathering analytics on utilization, where things are failing, um, what type of questions users are asking and queries users are asking.

You want to collect that type of analytics and build in those customized flags. And so we have these real time detectors, for example, classifiers that we build for our customers, and that they can customize that will go and actually find, okay, was there a 80% of of your LLM outputs refusing the end user's, uh, UL ultimate question, right? And or is there a new type of question that's being asked from end users that your model wasn't prepared to answer for?

And so now I need to go into my vector store and populate it with more information, or I need to figure out a better way for, for the model to retrieve that type of information if it's already in there. So it's a constant kind of monitoring of your AI system and, and really looking at it from a product point of view, uh, and, and getting the information you need in order to make, you know, adjustments that continue to have consistent performance. There's an old wag that says, um, you know, it's one thing to be wrong, it's another thing to be wrong at scale.

Yeah. Um, With this AI approach, we're seeing multiple LLMs and hec, there's even now small language models in the mix, and then there's all these AI agents and they're all interacting with each other alongside of humans. So how do I keep track of whether or not something is actually performing as intended, or, um, is that just kind of, I have to wait for somebody to kind of discover it and send me a nasty note or something?

Yeah, and I, I, I think, so that's, that's a fantastic question, right? Um, how do I know if the model or the AI system the product is, is performing as it's intended? And I think the key part of that sentence is as intended, but what does it mean to be, uh, to what, what, what do you, what's the intention behind deploying this AI system?

What is the user going to extract the most value for really defining your use case and what does success look like, right? And that's actually gonna be customized for each different organization, how they view success, right? How they view is this AI system being used for its intended purposes.

Um, and so like, I think this kind of goes back to that benchmarking or evaluation type of question, is each enterprise actually has a custom kind of definition of what they view as success for their use case, right? Maybe you want to, you're in a highly regulated sector and you wanna be really careful about the types of questions that you answer. You wanna make sure you answer it in a very compliant way.

We have a, a customer, for example, that defines a very unique term in the financial services sector, a certain way that the model doesn't actually refer to it in that correct context, that's considered a hallucination, or that's considered a failure mode, right? But that's really specific to that particular customer. Or maybe you're generating, generating reports and it has to follow a certain style or include certain set of references, uh, uh, et cetera, right?

That's the key thing, is to define what is your, what is success? And then build in very robust and repeatable testing for you to monitor. Is the model being able to, to, or the AI system able to meet that, that definition of success?

And the key thing here is we do see today across, you know, solutions, uh, not a lot of capabilities that are provided to the end users, the enterprises, for them to actually define success according to the very, very specific requirements. You'll see, you know, when the, the newest, uh, foundation model comes out, people are constantly benchmarking it on coding tasks, on multiple choice tasks they're giving at the lsat, et cetera. But is this actually relevant to the intended use of your particular use case, right?

Maybe it's really good at, um, ACEing the lsat, but maybe it's really poor at debugging your code, right? Uh, or, or generating your particular, you know, uh, transforming cobalt code to, to, to python code, right? Um, so, so you wanna make sure that whatever you're evaluating, you're, you're, you're explaining what is the intended behavior of the system and measuring against that.

One of the challenges that I hear about is that, well, the gen AI is probabilistic in nature, and it doesn't always do the same thing the same way every time. And a lot of the business processes that we're kind of trying to apply it to are, shall we say, deterministic, and people expect them to be done the same way every time. Um, so where do we kind of figure out it makes the most sense to apply a gen AI model to what kinds of tasks that are actually gonna, um, augment a process rather than, I don't know, break it?

Yeah. So, so there's certain parameters that you can play around with to try to get more deterministic about this. You know, there's determinism, there's also explainability.

These models may never really ever become truly explainable, uh, to the level that, that people hope they, they would be, right? Um, because if you can truly explain it, maybe you can have a more deterministic model, but let's take the determinism kind of point, um, so you can, you know, adjust the temperature to, to adjust for your particular use cases. In some cases, you're gonna want the more creative model, and those are the applications personally that I'm really, really excited about, is like, if you prompt the model multiple times, it's actually gonna give you new ideas of potential ways of reframing the problem, right?

And it's gonna be more creative and expand our creativity. I think generative AI has enormous opportunities in that space, but then generative AI also has opportunities in spaces where there's highly repeatable tasks, and you just need to get it right and have the right formula for how you do this. And that formula could be pretty dynamic and, you know, maybe rules based it, it may be more com complicated, but you wanna have that repeatability, uh, and, and robustness built into your system.

And so there's kind of a, a number of things that you want to do if that's the case. If, if you want to have a more deterministic and robust model, A, you can adjust the parameters of your model and make sure you have greater control over the setup of your AI system. You can't always do this with, with every model, with every AI system, but you also wanna have, again, those constant checks and controls, right?

So if some folks call this guardrails, uh, that's, that's what we refer them to, but you wanna be able to constantly monitor the AI system. And this monitoring can become really challenging as you go and scale as you have, you know, in some many cases we have customers with millions of queries every single day that their system's being hit with. So you need to build an automation to automatically check the behavior of the, a AI system and flag.

If say, you have agen workflow, that's kind of going outta control. Uh, the, the agents are, are kind of, you know, spiraling outta control. You want to be able to flag that type of bent and have a, a human review.

But you need to do this really, really efficiently because these, these AI systems today are scaling to huge, huge, huge workloads and volumes. And so you need to have a system that's gonna automatically flag when, when things are going awry and things are not going as you expected, right? That, that non-determinism is really showing, uh, its space.

And, and you want to be able to efficiently triage that and have it reviewed by humans When, uh, you can't manage or govern that, which you cannot see. So how do I get the observability level that's required? I mean, we have observability in DevOps types of workflows, but I'm not sure we have observability around ML ops and these AI deployment models yet, or is that kind of where we're headed?

Yeah, uh, definitely I, that, that, that's where we're headed, right? And so, um, you know, it's a little easier for these homegrown solutions that are building off of, say, you know, the, the standard list of foundation models. Um, and they're kind of composing their own vector database together and, and, and building this from scratch, right?

It's a little more challenging to implement the observability. Now when you have all your different vendor applications that you're using for, let's say, like, you know, SaaS applications or your document processor, et cetera, that are now integrating AI features into those products, right? So I think your CRM think about your, your, your document processor, um, you know, even your, your, your browser, et cetera, might all have AI capabilities built into it now, right?

And that becomes much more difficult to extend observability. And, you know, we talk to CISOs all day every day, and their big concern is they don't wanna look at 30 different dashboards across different applications and try to hunt out for where there's, uh, basically shadow ai, uh, being implemented across the organization and, and vendor solutions that updated with a new copilot in their last patch release, right? So what we've actually found to be really effective is actually a lot of these solutions, these vendor solutions, arping, operates to the browser.

Um, we personally built a browser application so that you can go and plug in, uh, your guardrails and your observability across all these different vendor solutions. I didn't have a single pane of blast to view this, so this is something that, you know, we realized was a problem. Uh, uh, many of, uh, of the folks that we talked to, uh, we very worried that they were not having control over those vendor solutions that they were bringing in house, but they wanted to see all the LM inferences that were going through there.

If there was a user that was violating their, their company policies, but they were using a CRMs copilot, um, they want to have that visibility into that application as well. To your point, we've always had trouble trying to manage all the dependencies that exist within our existing software. Is that about to become exponentially more challenging in this AI era?

I think there's definitely going to be, uh, additional dependencies that, that you have to look out for, right? Um, there's a lot of creative approaches that are, um, are coming out today. Uh, so, you know, one of our, our partners that's just, uh, a notebook, OEM, uh, you know, launching, uh, different capabilities that can embed a lot of these actually out of box on your laptop device, uh, and, and have kind of, uh, actually a, an agent that goes and, and scans through, uh, the LM interactions and make sure that, that they're being guardrail and observed, right?

Um, so there's, there's different creative ways, uh, actually leveraging AI to, to make it more scalable to do this type of work. But certainly, I, uh, you know, uh, I'd be remiss to say that, that there is a new ecosystem of, uh, kind of products and components that are coming together that enterprises need to be prepared for, uh, and, and integrate them together, right? They really, I think one of the, the key things here is the more that we can get the enterprise to, uh, or, or build a product rather, or a solution for enterprises to have a single pane of glass, uh, across all the applications for managing governance, for managing observability, the better these enterprise is gonna be.

Uh, but you do see, you know, um, different kind of platforms emerge around different hyperscalers, et cetera, uh, that they're integrating a lot of these opportunities, uh, but it's hard to bridge this across cloud environments as well in a kind of multi-cloud strategy. And so you want something that, that can kind of, um, uh, uh, reach out to all these different components and, and kind of integrate them into a single place. So when we eventually wind up creating AI models and AI agents to govern AI models, and we're gonna rely a little bit on AI to manage the ai, is that where this might wind up?

I believe there's gonna be a big component of that, right? And especially for regulated industries, there's always going to have to be human in the loop with this, right? Um, and, and this sounds like a new idea, but actually if you look at some of these really heavily regulated industries like financial services, they've actually to some extent been doing this for, for quite some time.

You'll look at anti-money laundering, uh, su suspicious activity reports, right? Or in the fraud space, there's a first wave of automation that comes in here that goes and tries to find the suspicious activity and then flag that suspicious activity for a human in the loop to review, right? Uh, and, and it's all about how good your automation is, how trustworthy your automation is, and you do have to continuously evaluate whatever tools are, are being used as at that, at that automation layer.

Um, uh, frequently people call this, uh, space, you know, uh, LLM as a judge. So, you know, LLM judging the outputs of another large language model, right? And, and potentially correcting those outputs or flagging or non-compliance or failure modes for hallucinations, et cetera.

And that is a necessary component, I believe, for this to scale, uh, practically, right? But you do need to build people, people who shouldn't forget. I think too many companies, frankly, forget about the second component here, which is building the workflow for the human in the loop to do the forensic analysis they need on top of all that enrich and metadata that the LLM judge is generating.

'cause at the end of the day, that LLM judge shouldn't be the final say. It should be really enriching the metadata and distilling down the problems for the human loop, right? Um, and, and, um, so that, that's, that's really how, how we see this kind of rollout.

And again, a lot of the folks that we work with, the enterprises, they're hitting massive production scale today, uh, with hundreds of billions of tokens every year, right? So having a, a team of human reviewers go in and, and monitor these, these ls, uh, just simply is, is never going to fly. You do need to use the power of LMS the, to scale this kind of observability, uh, and what we call case management, but you also need to build the infrastructure to effectively manage those cases, right?

And generate cases and be useful and relevant to, uh, the, the subject matter expert or the cybersecurity professional who will be doing the, the further investigation. So what's the one thing you see organizations doing that, you know, still makes you shake your head a little bit and go, folks, we need to be a little smarter than that. Well, I, I think, um, most enterprises that we talk to today are, um, still very much at trying to figure out their overall governance, like how they're gonna set up, uh, uh, the, the, the governing structure for aod.

And this might include, um, you know, what is the role of the ciso, what is the role of, uh, model risk management? What is the role of the product owners and the business line owners and the CIO, the generat AI platform owners, right? We see a lot of debate across the board.

Uh, you know, I, I wouldn't say there's a, a one size fit all solution, uh, to, to, to solving this, but we see a lot of debate across enterprises that, that we talked to about how to properly set this up, which stakeholders should be in charge of monitoring, or let's say, let's say, of, uh, of, of implementing a defining guardrail. Should it be legal? Should it be cyber?

Should it be compliance? Should it be the product owners? Should it be a mix of all those, right?

And what we find is that those organizations that kinda speed ahead without figuring out this governance structure, ultimately run into bottlenecks down the road, right? They ultimately will have a showstopper where they're like, we don't know. It's not a, uh, that this problem can't be technically solved.

Let's say there's, um, certain level. Uh, but let's say that, you know, uh, certain, uh, uh, stakeholders within an organization are not comfortable with productionizing a use case yet until there's more controls. Well, who, who's responsible for building those controls?

Is it the product owner or is the, the CISO who's supposed to be recommending those controls? The more that organizations can understand how to set up their governing structure and which stakeholders are responsible for these roles and, and build really a robust process for productionizing these solutions, the better these enterprises are gonna be down the line. I think there's just a ton of work, not only on the, the technical side, but, but just on the organizational side, uh, that, that, uh, enterprises are still working through Folks here, human in the loop.

It's not just some sort of buzzword that we use to pacify people. It's actually the difference between success and failure. Hey, Christian, thanks for being on the show.

Thanks so much. Great to be here. ai.

When you can find this in other episodes on our website, we invite you to check them all out. Until then, we'll see you next. Do.