AI Surveys, MLOps and New Banking Services – Techstrong AI Podcast EP 12

Hello and welcome to the Techstrong AI podcast. It's another week and we have more news and information. I'm Amanda Ani, and I am with Mike Fazar.

How are you doing today? I'm doing great. Always happy to be here.

Well, we're gonna start off with some interesting news, um, from the Linux Foundation and, um, they, uh, have partnered with some other organizations to provide a new open AI platform. So, uh, what are your thoughts on this? Well, this seems to be the anybody but Microsoft or open AI and any closed platform association, right?

I think it's called the open platform for enterprise ai. OPI guess I'm not quite sure what the, uh, phraseology is gonna be for that, but, um, on the one hand it's nice to see that somebody's doing this and that they're kind of having some sort of discussion about the framework and maybe playing nice together so that we can reduce the cost of implementing all this stuff because it does take a small village when you add up all the piece parts. That said, I'm not sure at this stage in our lives that we really need another foundation consortium thing to drive a thing, because in my mind, a lot of the customers are pretty smart about this at this point.

They understand the need for open interfaces and standard interfaces, so I'm not a hundred percent sure that, you know, we're serving a particular need here. Um, and they know the difference between open and closed. And if this was 10, 15 years ago, I'd be like, yeah, we definitely need this.

But I'm not entirely sold on the concept of yet another consortium because well, um, we have too many consortiums as it is. Yes, and we're gonna, uh, dig into that a little deeper later in the show about the open versus the closed ai. So you're right to be, um, a little bit hesitant about another open ai.

So we'll get to that soon. But moving on, we have a very interesting 300 page survey from Stanford University. Um, it's their AI index report.

A lot of information you can, uh, read more about it and, and get a link to that full report on tech strong ai. But, um, some key points from that is that AI tends to be doing better or almost at the point of doing better than some of the things humans can do. And a few of those I can list for you right now, which are visual reading, multitask language understanding, and almost better at competition level math.

Um, now on another side of this survey, this is where we're gonna talk about this is open models are making adoption a lot easier and, um, it it's making things easier for companies to go that route, but closed models are performing much better still. So what are you hearing from business leaders and what are your thoughts on this? Well, all right, there's two things here.

Let's go back to the first conversation as it relates to this conversation right now. I'll just point that over the history of open source over time, the pace of innovation in the open source space is faster than in the commercial space or the proprietary space. For one simple reason, over time, more people contribute to the project, more ideas or float through the project.

And there's a lot more peer reviews. Any company has a limited number of resources that they can throw at something and they may have a, a short term innovative leap, but over the long haul, it seems to me right now that whether it's Linux or a dozen other open source projects, it always seemed like the open source folks long term have the advantage. As for the report itself and AI getting smarter, it's 300 pages.

I don't think it really kind of explain succinctly why. Um, basically as you open up the context windows and you have more parameters, the machine can understand relationships better. And so when you're starting to see early on is a lot of AI systems where entire series of tasks are being automated, such as go create me a website and it we'll know what steps to follow in what order to execute the task.

So the AI is definitely getting a lot smarter. I think over time we're all gonna wind up supervising these things and a lot of the mundane tasks that we do today, just remember, I know that half the people listening to this can probably write better than the AI can today, but the AI keeps getting smarter. And so, you know, this time next year you may not even bother writing and editing the piece as much as you do today.

You might shape it, but, and but to reflect what it is you were trying to say better, but ultimately more and more of the lifting is gonna be done by the machine. And I think that's what this index is kind of showing, is that these things are getting smarter and of course the compute power will get, uh, less expensive over time. And so the things that are very expensive today will be more affordable tomorrow.

Definitely. And I agree with you, but I do think we are absolutely a long way away from the age of singularity, as they say, where, where the AI just takes over and is that smart to be left alone. So, um, but an interesting report.

So again, if you wanna read the full thing, it's, it's on our website, um, And you, you make an important point. Um, so it is a long way off before that and these things cannot be left to themselves. And I think one of the, uh, natures of our humanity is that sometimes, you know, when somebody says something convincingly, we just believe it.

And what we have here today is machines are convincingly not lying to us deliberately, but they are hallucinating in a way that makes it sound like you know, what they're telling you is gospel truth. When in reality upon further review you might find out something entirely different. So just remember these things are probabilistic, they're making a best guess not deterministic, which means that they are not always right.

That is the truth. So we have another study, and this one's from Gartner. Um, this one was over, um, uh, several enterprises and it was about their use of LMS and um, and how much debt they're gonna end up technically their technical debt.

Um, as they start implementing this technology, uh, many of them may find that by 2028 they're gonna stop implementing the technology because of all the technical debt or because of complexities or cost. So what are you hearing from business leaders and what are your thoughts? Well, here's the challenge.

Not everybody's excited about being overly dependent upon an an LLM that's built by an open AI or somebody who's got a hardcore profit model around it because it's costly to use and you wind up over time getting yourself locked into these things. As we were talking about in that first segment, I think though what this is pointing out is that if your answer to that is to go build your own LLM, well that may prove more trouble than it's worth. So there are these so-called foundational LLMs that you can use to as a baseline for then extending them to create your own LLM for a specific use case within a vertical industry.

Or, uh, some folks are using them, for example, to create a security LLM that sits on top of the foundational LLM. Uh, so I think we need to, you know, think long and hard about absolutely when do you need to build an LLM because the pace of advance in this space is so fast that by the time you build yours and then you try and update it, you'll be, you know, five or six generations behind what the rest of the doing. So that's kind of what the study is talking about with technical debt that said, um, think long and hard about how much data you wanna expose to somebody else's LLM and how locked in you wanna get.

This is very true and, and um, I think it's a, a difficult decision for many business leaders when it comes to, to what route to go. And really it comes down to having, um, a clear goal and a clear end goal and really understanding what tools do you need, um, because we're gonna get more and more of this issue coming up with the data too, like you said. Um, and And a lot of it people have a bias towards build, right?

And that's 'cause they just love the toys and they love computer science. But um, this is as much a business conversation as it is a technical conversation. It definitely is.

Um, so next up we're gonna talk about an article you wrote about um, um, a another survey. This one is from Docker and we have lots of interesting surveys this week. So can you share a little bit about the docker survey as it relates to gen AI and developers?

Well it turns out the survey finds that this may not be that all surprising, but developers are using gen AI tools like crazy and they're using it for everything from generating code to creating documentation, to just figuring out how somebody else's code works through a summarization. In that regard, probably all good things. The issue again comes back to this probabilistic versus deterministic thing and a lot of times the code being generated might not work.

It might contain vulnerabilities and might not be nearly as efficient as you would like it to be. So in its current form you need to look long and hard at that code. And I know everybody's racing to get things done, but um, as we go along, uh, things will get better in terms of the accuracy, but right now it's still early days as far as these platforms are concerned.

So, um, I think that developers are kinda excited about the tools. I wonder if they're overly dependent upon it. And one of the things that came out in the survey is that, you know, at least nearly half of them thought AI was over hyped.

So they're increasingly gone from, wow, this stuff's gonna change my world to, it's all that There is a lot of face being put into ai. Um, and it can only do so much at this point. And I think we had a conversation too about um, the skill level required you really still have to have a high knowledge and skill level to even know how to use these AI tools at this time.

So, um, now down the road that might be different, but right now it seems like you have to have some level of knowledge for it to be even useful, Right? Sometimes you're so busy figuring out how to daisy chain a bunch of prompts together that you might as well have written yourself in the first place. Yes.

Um, so then we have another very interesting, it's kind of in the same um, realm, but um, George Hume, he submitted an article on Techstrong AI about ML ops no longer being optional, um, as it's expected to grow because it's important for collaboration and for acceleration in the enterprise. So what are your thoughts? Yeah, I really like this piece 'cause it points out a fundamental problem.

So early on when very few people were doing AI models, the data science team pretty much ran everything. But the truth of the matter is they wind up spending an inordinate amount of their time on data engineering and application development issues and all kinds of things that have nothing to do with data science. They're expensive.

And so I think what's gonna have to happen soon is that uh, folks who run the DevOps platforms or slash platform engineering teams and all those data management folks are gonna have to take over more of those functions. And I would argue that, uh, an AI model is a new type of software artifact. So the folks who are building the artifact don't need to be the same folks who are deploying the artifact or for that matter even updating it.

Um, I think we're gonna see more separations of concerns because if every application has some sort of AI inference engine or some sort of AI component in it, it's not gonna be feasible for data science teams to deploy all that. That's a Yeah, it may have to be divided up into a few more sprints among each one. This is exactly right because otherwise who won't, who can afford to pay, you know, 20 people who are so-called data scientists six figure salaries to go manage it end to end.

That's just crazy. Yeah, absolutely. So that brings us to our last topic for today, which is that Oracle has added a service, um, to leverage gen AI to help banks and this will help banks with, um, detecting fraud more quickly, stopping money laundering, um, and things like that.

And it's called the Oracle Financial Services Compliance Agent, which is an AI cloud service. And, um, you have a little bit more information about this, so share with our audience. Yeah, I like this too.

I don't think it's necessarily a brand new idea. I've seen other folks talking about the same thing, but Oracle may have the resources and the relationships to implement it with folks. So, you know, hopefully that will all come together.

But this fraud thing that goes on, we're not aware of it enough and, and literally billions if not trillions of dollars is being lost through these, uh, fraudulent efforts that span everything from, you know, let me take your money for 10 bucks on Facebook 'cause I didn't actually deliver the thing I just processed your credit card to, to, you know, major medical fraud that goes on that steals from Medicare. And then, you know, that's a multi-billion dollar industry in its own right. What we're getting at now is that AI will, uh, be able to discover the anomalies that show up in these claims and um, and and understand the relationships between different things better.

Will it totally eliminate from probably not. And maybe they bad guys will figure out something else to do, but the amount that we currently have is gonna be dramatically reduced, that's for sure. Yes, absolutely.

And of course, you know, threat actors are using AI in various ways and which is why a lot, uh, there's a lot more threats, a lot more technical threats that are harder to catch. So we're gonna have to combat AI with ai, right? And it's not like the companies who were victims of this stuff, you know, just eat those costs.

I mean, the truth of the matter is they just pass it it along to the rest of us. So I think we all have a vested interest in this. Definitely do.

Well, that brings us to the end of our news today. We will surely have a lot more for you next week, so stay tuned and have a great rest of your week. Thanks Mike.

All right, as always, a pleasure man. I talk to you soon.