AI Everywhere: The Benefits and the Flaws – Techstrong AI Podcast EP17

Hello and welcome to the Techstrong AI Podcast. I'm Amanda Ani, and with me today of course, is Mike Baard. How are you?

I'm good. It's been a, a heavy week of AI everywhere you go, but we're gonna dive into that, but so far so good. Definitely has been.

And we're both getting back from tech conferences where that was a big focus. Of course. So with that, starting out, you shared some information from the Microsoft Build 2024 conference.

Uh, specifically they announced a co-pilot for AI to help development teams, uh, the co-pilot for Azure. So can you share a little bit about this? Yeah, Microsoft, yeah, they got my headstart on everybody.

There's just no two ways about it. The challenge sometimes is just sorting out a little bit between, uh, things that they announced and the way that they parsed the verb is versus things that are still in a, uh, shall we say, private preview. Some things are in a public preview versus what's generally available.

So sometimes in, uh, some of the stuff they would just say, we're gonna do this, but they didn't actually say when or how. So. Um, but that said, you know, they're clearly, um, ahead of most folks still in terms of their use cases for generative AI and copilots are now essentially permeating the entire portfolio.

So we're starting to see those in, um, for the management of Azure, which is a good thing in the sense that it will democratize that a little bit for the mere mortal IT folks may not have to know as much about how to manage infrastructure as code, but as much as they did all that, there was also a few things in there where they were talking about AI for more average folks who were gonna go just, you know, take a document that they create and create or train an LLM to do that. And then they'll have the LLM go automate a process. So it felt like they were signaling that they were gonna take, um, AI well beyond just kinda, you know, let me help my, uh, office documents or help me, uh, get a coder to automate some things for creating, I don't know, a website, whatever it may be.

But, and the signal was clear that, you know, we're going well beyond just the, the normal base of folks and we're trying to empower end users to do things that maybe in the past required an IT profession. Yeah. And I think we're seeing that more and more from all the big companies that are trying to go beyond that original realm.

Yep. Um, not quite clear exactly how that dance is gonna play out, because, well, everybody's job functions are gonna evolve. I think everybody seems committed to the fact that everybody's gonna have a job, it's just gonna be different and the relationships are gonna be, uh, somewhat, um, tenuous still.

So we still need to figure all that stuff out. A lot of folks are, you know, not saying that we're gonna replace developers anytime soon, but there are other folks who are kind of like, you know, hell bent on, we're gonna change the world and see what happens. Um, I thought, you know, the, the keynote note from the Microsoft CEO was solid this year wasn't, um, too light.

And as they say, and sometimes in the past, I think there's been a tendency where there's so many things that they are trying to roll through that they don't give the proper level of, uh, do a chord. Um, so, you know, sometimes when I've been to that show it's kind of like, here's the long list of all the things we're announcing in full note, end of keynote, see you later. So there was at least some perspective provided the she, which, uh, I think a lot of the people who attended enjoyed You can cash it out online still and check out what I'm seeing.

Of course, there's, I don't know, I think they announced more than 50 products and I would say at least two thirds of them had some sort of AI component into 'em. So, you know, the race is on. Yeah.

And just, uh, from, from what I'm hearing, I don't see how, um, the developer jobs could go by the wayside anytime soon. Now they're certainly gonna be more efficient, um, utilizing these tools, but I don't think AI's anywhere near the, the ability to just replace the type of the developers. I mean, and I think that's the quarter it, right?

I mean, Microsoft is essentially saying that, um, developers will become 50% more efficient. Well, that's great, but efficient doesn't always lead to more code being written because well, code requires some level of inspiration at the time, and I have to have an idea, but at the very least, a lot of the toil that gets in the way of having a great idea should be reduced. So, um, I don't know if that means developers are just gonna have more time to play games or whether or not they're actually gonna write more interesting applications, but folks are promising an an application renaissance where there's gonna be more apps developed in the next two years than in the last 40, who knows.

Yeah, absolutely. Well, let's move on to the next conference. There were so many conferences going on.

Uh, IBM think was another big one, and, uh, you shared some information from that about a new platform called IBM concert, uh, and this promises to extend the scope of AI ambition. So can you share a little bit there? Yeah, I mean, I think IBM went from kind of leading the AI charge to kind of chasing after everybody else a little bit.

So this was an effort to kind of put themselves back in, uh, the fray as it were, in a, in a leading position. Um, concert is a essentially, uh, centralized IT platform that makes use of, uh, LLM such as granite to pull in data from everything. Uh, right now they're leading up with, um, you know, the core platforms they have from Red Hat and, and, and, and other places to pull it data into a way that makes it easy to centralize the management of it across a hybrid cloud computing environment, which, you know, is a pretty cool and a pretty ambitious use case for, um, gen ai.

So we'll see how that all plays out. I think it's still a little bit early, um, but, um, it's not clear to me that, you know, when I look at it today, you know, IBM went from, you know, king of jeopardy to what's starting to feel a little bit like an also ran maybe with, you know, Microsoft, Google, AWS meta and everybody else kind of banging the drum and leading the charge. So I, I thought the IBM think conference was a good step in the right direction, but clearly they need to do more faster.

Yeah, absolutely. Um, like you said, they led the charge Yeah. Last year with their big Watson X announcement and, um, you know, it, it's a huge race between all these companies.

Who's gonna be on top, who's gonna have the next best offering? Yeah. Or, or just make it easier to operationalize.

I think a lot of people right now are kind of like, yep, gen ai, we get it now. Can somebody make this easy? Mm-Hmm.

Simplify the process. So moving on GPU optimization, this is a big topic amongst business leaders and, and you recently shared an article about Sky Mail emerging from stealth to share a hybrid GPU platform for AI models. So can you share a little bit about that news?

Yeah, I mean, this thing is early days, but it's interesting tech. Um, so I felt it was worth calling out. There's a shortage of GPUs, we all know that.

So what they're saying is that you can split the inference model a little bit and only run a portion of it locally on an endpoint or in a data center, and that, um, it can then go invoke some cloud resources that it may need to run additionally. And this is relevant because it means that I can rightsize the inference engine to whatever platform that I happen to have. So I'm not trying to overload an inference engine onto something that can't handle it.

And I'm not stuck in a position where I'm trying to go find hiring GPUs that I cannot find in the first place. So this kind of area, you know, lets me kind of, uh, work towards a, a middle ground a little bit. There may come a day when GPUs are widely available and inexpensive, but in the short term, it looks like we need tools like this to kinda find a way to maximize the OR and optimize their limited resources we have.

And of course, a lot of folks are moving inference engines off of GPU to something else, but you know, then you sacrifice the parallelization capabilities that's inherent in the gpu. So, um, I think these guys are probably on something that's a good idea. I don't know if somebody else will do something similar, but let's, uh, let's say this is a step in the right direction.

Yeah, we're, we're hearing from a few smaller companies out there that are trying to solve this issue, but nothing seems really big at the moment. Yeah. Um, you know, in the meantime, NVIDIA is just racking up profits left, right, and center.

And so, um, congratulations to them. But you know, it industry abhors a vacuum and right now it seems like there's one here in the GP space. Absolutely.

So this is an interesting survey that recently posted on Techstrong ai and it's about women tech leaders appearing to outpace male peers in generative AI adoption. So, um, this survey showed that when it comes to adopting these tools that women appear to be, uh, quicker to do so and uh, and are looking for ways to be more efficient and, uh, therefore, uh, willing to try out new things and be a little bit riskier in their decisions, uh, which is coming out to, to help them as it appears. Yeah.

What are your thoughts? I have a pet, I have a pet theory about this, but since I am not the one of the survey respondents, I'd be love to get your take on it a little bit. I mean, what do you think is going on?

And we have a colleague, Bonnie Schneider, who, you know, can't get enough of all things Gen ai and she's like, you know, at the forefront of all that stuff and playing around with things. But what is your sense of, um, you know, is there a sort of a, a, a gender gap aversion here or what's going on? Um, I'm not so, uh, I'm not sure if it's a gender gap aversion.

I think, um, it has to, and in the survey it shows that, um, there are perhaps some areas, um, where women are just more willing to adopt this technology to help them, uh, if they don't understand something, uh, versus the men. So, um, and, you know, not as scared to try a new tool. So for me, I love all things AI as well.

I have the new version of the chat, GPT-4, oh, I have, um, various other things I like to try out, you know, anytime there's a new tool, I'm all about trying it out, see if it's gonna help me. Um, and I think both with men and women for that skills gap, um, that generative AI especially is helpful, um, to bridge that area and make things easier. Um, it just appears from this survey anyway that um, uh, women are more likely to try to use it to help.

All right. Well, mind Pet theory goes to the Barbie movie, which is made it pretty clear that there's a lot going on in the average daily life of a woman as she kind of had Trump balances all those requirements that are impinged on her by her, uh, family husband, society in general. So I think that there's a certain appeal around Gen AI that just says so many, so many hours in the day, I really wanna kind of maximize 'em.

I got a lot going on. And anything that kind of eliminates toil and reduces friction is gonna be, you know, I'm, they're gonna be at the front of the line. Oh yes, of course.

I mean, women generally are juggling a whole lot between work and personal and family. And so any tools to help make that easier is going to be embraced. All right.

Well there we are. It's not rocket science, it's just good old common sense. Absolutely.

And last on our list today is issues with trusting ai, which of course there's lots of issues around trusting AI and, uh, immersive labs uh, issued a report about some, a contest where they had to prompt an AI bot and see if it would give them secure information and almost, I think 80 over 80% were able to get secure information. And as it got harder, even with the harder tests about, um, uh, I can't remember exact percentages, but it was a pretty large percentage could even crack through those harder tests to get the AI bots that IUL secure information. So of course, this is a concern if it's that easy to get secure information, what can business leaders do?

Yeah, I think that we're gonna have to take a hard look at this. I mean, so if you look at the report, what they're saying is they've got, you know, 30,000 or more people involved and they said, you know, there's a secret somewhere in this, uh, LLM and people started just, you know, toying around with ways to kind of, you know, manipulate the LLM into giving up the secret. 'cause theoretically you're supposed to have some controls around that, but turns out people are pretty clever.

So, you know, sometimes they just ask straight away, you know, what was the password or the secret in other ways they were using essentially social engineering techniques to kinda tease it outta the LLM. And some people would do that by, you know, they would ask the LLM to help them write a poem that happened to be about sensitive data, or it would be, um, something involving a story or a narrative. But some of the ways they went about doing it were pretty creative.

And it just shows that humans are, have a certain level of ingenuity when it comes to these kind of things. And of course, we've seen the bad guys be pretty good at social engineering techniques, and if they apply that to ls, well, you know, they might be able to pull out all kinds of sensitive information that they're not supposed to have. So the report is just, um, you know, immersive labs is all about training.

Um, so they were just trying to highlight the fact that, you know, we need different ways of thinking about data loss protection and data loss leakage. 'cause that seems to be at play here. And I think a lot of companies will take a giant step back.

'cause uh, if you don't know exactly where that data's gonna kind of manifest itself and how it's gonna manifest itself, the costs for doing that are pretty high. And yet, as we now know, every end user is at home playing with stuff, and there's a whole shadow chat G-B-T-L-L-M process going on. So I got a bad feeling that this is gonna be kind of one of those incidents where it's only a matter of time before something very bad happens.

Yes. And I, I really am not sure what the solution would be. I mean, certainly it's proven that humans are much more intelligent than the AI at this time.

Uh, you know, so how we're gonna address this, I know there's lots of laws and regulations trying to be formed, but, you know, laws and regulations don't stop things from happening. So, you know, that's an Issue. I think it may have to come with, you know, like a little thing that says, you know, a label says use at your own risk.

'cause um, I think we tend to trust these services more than we should, both in terms of, you know, the output itself, but also, um, you know, how secure they are because, well, we think that, you know, there's these big companies behind it, but I think we're all kind of rushing down this road without necessarily, uh, thinking through all the subtle nuances and, uh, we're gonna skin our knees. There's just no two ways about it. Yep.

Well, that does it with our list of AI topics for today. com is where some of those articles were posted, and that'll fill you in on the gaps. Meanwhile, have a wonderful week and thanks Mike.

All right folks. And as always, all things ai, be careful out there.