Techstrong Gang – February 20, 2024

Hey everyone. I'm Alan Shimel, CEO of Techstar Group. You know, you, you can't walk three feet today without tripping over something, ai.

Um, but how real is it, especially in it, what are CIOs need to know? What do developers need to know? What do you need to know about ai?

We're gonna be discussing all this some more today on Textron Gang. All right, so we've got a, an AI Power Pact Show today here on Textron Gang. Let me quickly introduce the gang if you haven't been following along.

To my left here is our Techstrong Chief Content Officer and all things AI for us today, Mike Ard joining. Mike and I not here in our Bo studio is our tech strong AI editor, Amanda Reini. Amanda, welcome and of course, uh, from Way Out in Denver, Colorado is our, uh, CTO and Chief Research Officer at Techstrong Mitchell Ashley.

Hey, Mitchell. Welcome. Thanks Alan and our special guest start today.

Another one of the gang members. He has become quite the AI expert. Uh, you may know him from his newsletter, which is widely circulated, the artificially intelligent enterprise, but he's, he's much more than ai, even my friend Mark Hinkle.

Mark, welcome to the gang. Thanks, Adam. Alright, Mike, why don't you kick things off today?

Well, we have a story this week up on Techstrong AI talking about how 2024 will be the year we operationalize LLMs large language models for the uninitiated. And I guess I wanna start with a chat with Mark, 'cause I know you were here for the hackathon that we had. Uh, I think that was two months ago now.

Um, it was More, I think it was August. Yeah. Mm-Hmm.

And, and what are you seeing from folks or hearing from folks about operationalizing this stuff? 'cause I feel like last year was the year of irrational exuberance and this year we're getting down to some work, but it's not clear to me we actually know how to do this. Yeah, I think you're right, Mike.

I think, uh, you know, last year was the year of the AI pilot. By the end of this year, it'll probably be the ai uh, the year of scaling and ai. And I think it comes down to a couple things is, um, first one is the expertise to do it.

The second thing is really the security around data going into it. So data privacy is at the top of the list for people. So I think the low risk, high rewards, um, you know, customer service, things like that, they're already public, is where people are starting.

And then, you know, the third thing is we're trying to put this stuff in production and every week there's a new LLM or small language model coming out. So picking a vendor, picking a technology is really tough. 'cause every year there's a new, or every week there's a new shiny toy coming out.

So that's sort of the three, you know, buckets I look at is, I think we are gonna see it, but low risk, um, being able to do so securely and then picking a technology that is best of breed and stable enough to go into production. Yeah. I can't figure out whether or not people are gonna actually build LLMs or if they're just gonna take something like a Vector database and expose some data to it and try to customize it that way.

It seems like that's the path of least resistance. But that is an LLM, but that is building an LLM, right? You take your unstructured data or whatever, you take your data, you put it in your vector database, you feed it in as an LLM.

And now it may not be as big as the, you know, the set that chat GPT runs on or something, but it's an LLM nevertheless. But as Mark said, maybe some of them become small sl, small language modules, s SLMs, right? Um, but here's what I think's really interesting, just like everything else in it, sooner or later, it comes down to the data, right?

It's all about our data. And I think data privacy is, is, is a huge, uh, you know, not knock at the door this year for, for ai. And, but I do think that this is the year where a lot of people will create their own LLMs s SLMs or whatever you want to call them.

I think people still think there's a gee whiz wow factor in that. Mitch, you agree? Yeah.

I I I, I do, I think that, you know, we were talking on another Textron gang about, uh, the vector databases kind of being the entree level for it, as opposed to trying to go build an LLM, which is much more complex that that is behind the, the vector database. It seems to me, and there was a good analogy in the, in that article about we're kind of going through the cloud evolution again. Oh, it's very costly.

Oh, how do we secure it? Oh, what about our data? Oh, uh, there are 50 million options.

What do we choose? What do we standardize on? We're kind of going but, but in a much more accelerated pace with ai.

So maybe there's lessons learned from cloud, I don't know, but it's, uh, deja vu all over again. Amanda, what's your sense of what's going on? I know you and I have a separate podcast and we were just talking about how frankly, this AI stuff is shining a light on the fact that we kind of suck at managing data and have for a long time.

And, uh, we've been hiding this issue from folks and now we have to go clean up our mess. Yes. And it's really hard once you get unclean data to get it outta there again.

And the issue with all these LLMs is there are so many choices. All the business leaders are really overwhelmed with the number of LLMs and you know how to use them. And they really ne need more, um, specific tasks, not these generalized LLMs.

So figuring out how to, um, get more specific tasks done by smaller models and then clean data in those. So look, this, this is, this is an issue we saw when we were doing the interviews. If anyone had a chance to see what we did at, uh, AWS reinvent.

Mm-Hmm. I think the better companies are creating a middleware layer where you could plug, you could plug any LLM you want virtually in there, right? And that's the beauty of that kind of system because just like we saw with cloud, right?

Remember everyone said, oh, hybrid cloud's a no brainer, but I don't know who would use multi-cloud. Why complicate your life like that? Well, the fact of the matter is, as the different cloud providers have evolved, each one does certain things a little better, right?

Um, Azure has some great functionality that maybe you don't quite get with AWS and Google has something you don't quite get with it either. The other two, I think the future is, I can plug in different LLMs. I I'm not, I'm not signing up for the apple walled garden model here.

I, I want an open structure. I can engage and disengage, plug in and unplug LLMs as I need them and as I see fit and I, I'll use the right LLM for the job. Mark, what are you hearing from folks out there?

'cause you've been a little quiet, so I'm gonna call on you. Yeah, I think, you know, just sort of tying together what we're talking about is, I think what we're gonna see, and Alan struck on it, is there's probably gonna be a layer that Ralph's conversations based on the right expert. So let's just say each one of these LLMs has an expertise.

So for general stuff, we may use a general LLM, like one coming from open ai. But then what we will do, and when, going back to the training question, is when there needs to be a deep domain expertise like financial or medical, those are the places where you'll train. So, you know, Kaiser Permanente is a big health provider.

They're gonna train their own private model. Bloomberg, um, has their Bloom, GPT, which is very much trained on their a hundred years of data. And what'll happen is based on some set of middleware policies implemented by today, uh, I just saw, um, Kong, one of the open source API gateways released a new project that actually will, you know, route to the doctor, route to the financial analyst or route route to the generalist based on that.

And in that case, then when they go to the generalist, they'll take that data using, uh, um, rag or retrieval augmented, um, generation and pull your data from the database to make it more likely to be, um, accurate without having to retrain. So that's where I look, I see things I, I'll tell you, I, you know, so I think the action in this upcoming year isn't going to be on the ability of people to take unstructured data into a Vector database and create an LLM or SLMI think that becomes commodity, anti non anybody could do. I think the real issue is just like we said, prompt engineering is becoming a, a job, um, training engineering, right?

LLM training engineering's where the money is, right? Because I could, I could use any old AI to go create some, take some data and put it in a Vector database and spit out an L-L-M-R-S-L-M training that machine or that system to optimize it for my employees or my organizational use. That's where the money is.

And I think that's what we're gonna see people who know how to train these models. All right? Speaking where the money is, we have a separate article on the site talking about, well, CIOs are all for ai.

It turns out, um, I have a slightly cynical take on that, so I'll just start it out with, uh, you know, it seems to me CIOs have been feeling a little left out on the whole digital transformation thing for the last few years, and AI puts them back at the center of the thing, because as we were just describing, there's a boatload of data that has to get managed and kind of orchestrated, and somebody's gotta figure out how all this stuff works. So, mark, what's your take? Or are we gonna see, you know, CIOs back in this conversation?

Or is it all in the realm of the data scientists and those guys are still sitting on the sidelines? Well, I think the, they definitely, um, I think 25% of new IT budget projects are allocated to ai. So their buying power is, you know, pumping up and they have a mandate from their boards and their, their C-suite to, you know, their peers in the c-suite to go out there and capitalize on this trend.

But, you know, the, the vendors that are providing that outside of Microsoft in the AI space, they're gonna have a whole new crop of vendors to actually, you know, adopt things with a lot of maybe IBM as well. But, but the things that are cutting edge are, you know, I think are gonna be something that slowly creep into the enterprise. But I think they have the spend, I don't know if they have the knowledge.

And this is, you know, this paradigm of AI is a lot different than Cloud Cloud was just taking what you already had and putting it on someone else's computer. AI requires a lot more machine learning, a lot more, um, expertise and stuff that they're not used to. And a whole new crop of security problems, which Alan is a security, the space well is, you know, you understand how things go into your servers, but a lot of this ai, gen, ai, um, infrastructures black box.

So I think they're gonna, you know, be challenged to do it. Well, Co couple of things there. First of all, let call me cynical, but let's be real about why CIOs are so hopped up on ai.

Number one, they got budget to buy stuff and they love buying stuff. Number two, though, they got their ted's, a lot of their head count was chopped, right? A lot of budget in other areas were chopped.

And it's not just the CIOs, it's the CMOs, especially the salespeople. They're all, they're all have less head count, less resources and have to produce more output. And they see AI as, as the, as the golden ticket is the golden calf here, right?

That's going to get them, you know, bridge that miracle layer of doing more with less. And that's what's driving this whole thing. When we look at security, though, I think right now most people who talk about AI security are talking about data, right?

Securing my data that AI doesn't suck it in and make it available to other people. I think one of the things we're gonna see this year is as AI gets more involved in developers and in the development and the pipeline of CICD and software development life cycles, we're gonna need to worry about AI security risk and the software supply chain. And that's gonna be a huge thing, Amanda, is this just a bunch of old guys being too cynical or what?

No, it's absolutely the, it's absolutely the concern. It all comes down. It all comes down to it.

Go play in front of your own house. Get off the lawn, right? Yeah.

Get off my lawn. No, but I, I think that's what we're looking at, you know? All right.

We talked about security briefly, and I want to talk about a move by Microsoft, Google, apple, and others to join something called the AI Safety Consortium. And I gotta say, I was a little concerned about ai, but now that I see that Microsoft, Google, and Apple are actually working together on anything, I'm terrified because, you know, normally they hate each other. And so it must be really scary if they gotta come together to figure out how this stuff works.

So, um, mark, what is your sense of all the noise out there around safety and ai and of course, it's election season, so we're worried about misinformation and all kinds of stuff, but what's real and what's not real here? Yeah, I mean, I, I read the article and I, like you said, I was equally terrified when you get those folks in a room trying to collaborate. But, um, um, the thing is the, that the actionable thing that they were talking about is watermarking.

Well, there's tons of ways to do DeepFakes without watermarking, and it just reminded me of the old saying that if you outlaw guns, only outlaws will have guns. And so, you know, I don't know if, you know, if you're really sophisticated, deep fake, or you're probably using something that's already out there and open source and training your own models to do, you know, whatever, you know, hygiene you, you care to. So, um, I think it's great that you surface the issue, but I don't know what they can do that's actionable and will enforce, you know, other than they're all gonna agree to make it hard for you to use, you know, apple or Adobe or any of the major tools to do deep fakes.

But, um, I don't know if it has any teeth. I, I don't know if we have any teeth in it either. But here, here's the thing, and, and this is a almost uniquely American thing, right?

When something goes wrong or someone gets in trouble, they go to rehab, I don't care whether it's sex rehab or drug and alcohol rehab or whatever, but they rehabilitate themselves. I think these hyperscalers are trying to head this off at the press and starting a rehab, you know, 12 step program for AI security here. And this way.

Don't worry, Mr. User, 'cause we're all in rehab on this, and we're gonna, and we're working together to make it happen. You know, what will actually come out of it?

I, I don't have high expectations, but I, you know, the, the question is how, how real is the threat, Mitch? I think that this is just, uh, an attempt to head off the government before they start showing up with regulations. And I think it was Ronald Reagan who said the nine scariest words in the English languages.

Hi, I am here from the government and I'm here to help. So are we just trying to get in front of that or what? I don't think Ty was in there, but go ahead.

Yeah, but this definitely part of it for sure. I mean, that's why, you know, the, uh, the tech companies are engaging with, with the regulators and saying, let's work together, right? 'cause they know they're gonna run off and do the wrong thing.

I, I think the, I'll take the non-cynical view of it for a moment. The the upside of this is this consortium is housed within the, uh, within nist, uh, s AI Safety Institute. So they have their own kind of organization in nist and, and nist, NIST is pretty reasonable organization to work with.

They come out with good frameworks and standards. I'm not saying they're perfect. Um, and by the way, there's also Cisco and IBM and Intel and Adobe and Amazon and hp, and God knows who else in this consortium.

So they'll all kind of duke it out with the Microsoft and Apples the world in this. So we'll see if anything comes out of it. There's my skeptical part.

Amanda, what's your faith in government? Like these days you're in Texas, what are you hearing? Well, my biggest question is, yes, we need rules and regulations, of course, but you can have all the rules and regulations, but are bad actors gonna adhere to those?

So h how are we really gonna, you know, save ourselves from the risk if bad actors, but they're not really gonna follow the rules or the regulations. So, but, but once you have a rule, and you know, this goes to the rule of law. You, you can't prosecute someone for violating a law until there's a law.

So now this will create your, your laws, so to speak, your regulations, your compliance mandates, and then those who don't meet it, the question is, what, what could you do to them? Do you ban them? Do you shut 'em down?

Do you find them? Et cetera, et cetera. But I, I will tell you something, this whole argument of doing it before the government does that worked when we lived in a very different world where the, it was the US these days.

The real action, the real government bite comes from across the Atlantic. And what the EU does, and I don't care what all these other companies do as a consortium, the EU is going to wind up setting some standards here that everyone is gonna be, have to, uh, yeah, adhere. They've already passed their, well, they got their first ones in there.

Yeah. I just think there's a bunch of folks in Russia and China looking at us and having a good laugh going AI safety. It's never right.

Yeah. Well, and that's the whole point. You know, the, you know, Amanda's point about, and the outlaws who have guns think, look, they're gonna be rogue states and rogue players who, who plowed it, but does that mean we cut off their traffic?

I mean, the EU knows how to put bite into their rules, right? And y you hope for the best. I I don't have a lot of confidence in our government putting some regulations in place.

All right, so Mark, I wanted to get your opinion on this other story. Probably maybe the last one we get to. Okay.

Um, and I laughed when I read this story, I have to say, but maybe not for all the wrong reasons, but, so open ai, CEO Altman wants 7 trillion to go build AI chip plans, 7 trillion that is larger than the GEDP of half the countries in the, on the planet. And if he wants 7 trillion and everybody else wants 7 trillion, we're looking at what, 50 trillion to go build AI chips. Do we need these things?

What are they gonna do? And what's real here? So I, I latched onto it, and his original statements sounded like he was going to ra try and raise that himself.

What he's talking about is create an ecosystem where chip production for AI is, um, takes, takes it out of the hands of few and puts it in many. So it's an ecosystem so that NV Nvidia isn't the only arms dealer in the, uh, AI race. Um, he's, he's already invested.

He also has a big invested in, I believe is the, um, chip manufacturer. Mm-Hmm. But I think he's talking about is creating a supply chain that doesn't have a choke point because he sees the value in ai.

Um, um, he's probably also seen how close he is to, um, artificial general intelligence, the super intelligence that's higher than humans. And he realizes they put out a paper in the fall that said basically that even without data, you can get, um, leaps forward by compute power. Yep.

So I think what he's trying to spark is investment in chips beyond Nvidia and a few players like a MD and Intel to have a broad availability of silicon that can, um, do AI training and serve AI inference and other things. You know, I learned a lesson a long time ago in tech, very few things are revolutionary. They're evolutionary, and those who don't learn history are destined to repeat it, right?

When you look at cellular devices and the chip manufacturers qualm, Qualcomm became a, a defacto monopoly. Nvidia has the same potential here. I think he's trying to head that off.

I think there's also, it's not just Sam Altman. There are national security and national economic reasons behind this whole push. We don't want those NVIDIA chips or the next generation GPUs being made in Taiwan, where they're under the sword of ES or whatever, however you pronounce that Greek fellow, the sword, the sword of China, right?

Who may attack or not attack or whatever. We want those chips built in the us The whole Biden infrastructure thing set aside a lot of money to do foundries here in the us We want to develop, if, if GPUs are gonna be the next hot chip, we want 'em built here. Not in Russia, not in China, not even in Europe.

We want 'EM built in the us and I think that's a big driver for this whole initiative. I'm not sure if people in Europe care if they're made in the US or not, but Mitch, I'll ask you this question. I think it takes, uh, if you ordered a GPU today, you might be waiting till Christmas maybe, if you're lucky to see it.

So the question I have to you is, you know, are we overly dependent upon GPUs technically anyway, and for these things were built for graphics and we're using them to process algorithms, and maybe we can have chips that are optimized for algorithms. Well, there's you, you're right. That's where we started was with the kind of graphic GPUs.

There's an nvidia, for example, has H 100 now the H 200 model of their chip, which is designed for ai. So we're seeing ai, GPUs, if you will, um, that are coming out. And they're not the only ones.

Um, so, but, but obviously they're going to be very expensive. And I think that's our issue. We've gotta get to a point where we reach that, uh, economies of scale and competition where those things aren't, we're not stuck in the Intel game.

We're, you know, the CPU is the thing that we're kind of beholden to and we're paying, you know, $50 a computer to be able to have that one chip in there. So I think that's, that's part of, I I read Al Altman's article, the article about him, and I thought, well, there's a couple things going on here. One, building factories in the US that's gonna get appeal is gonna get government support for doing that.

Two is from a, also from a national security standpoint, right? We're getting all of our, now we're concerned about AI and the impact there, but, um, it, it, it's still early. I mean, it's expensive to buy any kind of a GPU, especially an AI one.

Yeah. And by the way, when I say GPU, that's just because what we call them today, right? But I think the whole point with Sam Altman's thing, and, and the whole push behind this is next generation AI chips.

I don't care whether you call 'em ZPU or UP Zs, their next Gen API chips or AI chips. Yeah. And the history of it would show that increasingly things that start up at the top end of the stack eventually push down into the chip anyway.

And so mm-hmm. The performance of these things will get better and hopefully they might generate less energy, maybe and be a little friendlier to the Environment. That's a whole, that's a whole nother issue in Mark and Amanda weigh in.

But, you know, all these GPUs, the carbon footprint in, in making them and running them is, is extreme, right? So you gotta start, 'cause the other common trend we've always seen in technology is what starts out on hardware eventually makes its way to software, right? So can we build some of these things onto software?

Can we reduce the carbon footprint? We're gonna need to, if we're not gonna boil here, like the frogs and the soup. Um, so, you know, I, I'm, I'm hoping with next gen chips, we, we see that kind of progress.

Amanda, I know you published and written a couple of stories on text drawing AI and some interviews around this issue. So what's your sense of, you know, is it a real issue or are we just kind of saluting it as we drive by and, but we're not really, as the frogs not paying attention to the fact that we're boiling? No, it's definitely an issue.

And of course, we talked about this a little on our podcast, um, that there, there has been a consortium formed to go over the energy output and the carbon footprint. And, um, aside from just the energy output is, um, the technological waste as, as, um, other equipment set aside. So, um, it's certainly an issue and we need to figure out, maybe we could use AI to harness the energy somehow in another way.

So it's not going out into the atmosphere. Um, but, you know, using AI to, to solve the problem. Right.

Hey, um, I know we're working with Mark on an upcoming shindig here. We will touch on a lot of these issues. Do you want to get into that a little Bit?

Yeah, sure. Actually, we gotta start wrapping up, Daniel. Anyway, we're coming to the end of the show.

If you love AI or even if you're just curious about AI, or even if you hate ai, I've got an event for you. Uh, we're partnering with Mark and the artificial, uh, intelligent enterprise, uh, to do a, well, it's actually 24 hours, but it, it's a 12 hour segment that'll be repeated as we kind of follow the sun, if you will. And it is probably gonna be the largest AI event to date where we are.

It's, it's free to register. We're looking for more speakers now 'cause we may have a hundred speakers or more. When all said and done, uh, we have sponsorships available as well.

Mark, I I don't have URLs and stuff in front of me, but I know you, do you want to tell people where they can go and if, depending what they're interested in? Yeah, yeah. Uh, we're gonna be live soon with it.

online. So the AI enterprise do online. Mm-Hmm.

And I'm really excited about it. 'cause I think that there's lots and lots of tech news out there about ai, but it's one of the first shows that's really about people who have business problems and how they can solve them using ai. So to Mike's earlier point, you know, people are stuck and they're stuck because they don't have enough access to knowledge.

We're gonna have some amazing speakers from, you know, the end users, vendors, researchers that are, they're gonna fill in those blanks for them. Absolutely. And then on the AI vein, I should also mention, if you're attending RSA this year in San Francisco at Moscone Center for the eighth year, we'll be putting on our DevOps Connect DevSecOps event.

But this year, of course, with AI being the star of everything, it's DevSecOps in the world of generative ai and we have some amazing speakers lined up for that, including, you know, uh, top level security folks from open ai, uh, uh, men's Google, what's the other one? Meta, meta Meta, uh, uh, not Eds. It's with a name.

Anthropic Anthropic. I'm terrible at names lately that that one's a tongue twister sometimes. Yeah.

And then it's hard to get that in. And David Bryn, who's a great futurist, Hugo Ne award-winning sci-fi author and PhD over at nasa, JPL is gonna be high headlining, also giving out his books. There might be a Mark Hinkle, uh, spotting there as well, or John Willis or you know, well Mitch, you're doing a panel there too, aren't you?

I'm doing a panel there too myself. Absolutely. Absolutely.

So that's May 6th. com. online as well.

So guys, if that's it, does that wrap up? That that's it. I just wanted to say that everybody here was actually here and is not some fake AI thing that we put in their face.

How do we know that? How do I know? Is that really Mark?

Check the water mark. They call me Deep fake Uhhuh. Alright, that's your online handle, huh?

Just a reminder, uh, if you're watching this live, we're tomorrow is, it'll be Wednesday and we don't have a fresh show, but we'll be back Thursday with some great more original content here on Techstrong tv. You could check Techstrong TV out to Techstrong tv. You can go to our YouTube channel, which is also called Techstrong tv.

ai, digital CXL, or LinkedIn, or even Facebook. So there's plenty of places to watch us. This Alan Shimmel for Techstrong.

Have a great day everyone. Thanks for joining us on the Gang.