Techstrong Gang – July 29, 2024

Hello everybody. I'm Mike Baer. Today we're talking about who moved my AI cheese, followed by the need for maybe a little more AI strategic planning.

And then finally, we're gonna take a look at this new large language model from Meta. You're watching Textron Gang. We'll be back in a minute.

All right, folks, and we're back. And let's talk and get to our guest today. We have John Willis joining us who has been, I don't know, John, you're probably the most involved person in AI that I personally know anyway.

But John, welcome. Hey, thanks, Mike. Yeah, great to be here.

All right. And then we have Amanda Razani, who's the editor of Techstrong AI and digital CXO, which is just brimming with AI copy every week. Amanda, how are you?

Doing well, thank you. And finally, Bonnie Schneider joins me today in the studio and our resident expert on all things climate. And you know, I gotta say, you know, you're coming up to speed on this AI stuff pretty quickly too, so thanks, Mike.

I'm enjoying it. This is, uh, becoming quite the core. All right, let's jump in here, folks.

There's a couple of reports out, and I think there's three different surveys, and I'm not gonna get into a lot of details on each one, but, um, the thing about them all is, it kind of says that we don't really know how our IT jobs are gonna evolve. One suggests that, uh, software engineers have embraced these tools immensely, but they're using them in, um, with mixed success, we shall say. And a lot of folks are trying to figure out, well, um, is this better for senior DevOps engineers or does it eliminate the need for junior DevOps engineers?

Another one now says, there's these folks called system administrators who aren't really programmers per se. They tend to use more graphical tools, and they seem to be struggling with understanding how to use gen ai, but there's an argument that says that they might ultimately benefit more because, well, the gen AI can create snippets of code for 'em, and they might not need as many software engineers. Um, John, I know you've been kind of following this whole space, but let's jump in here.

Where do you think all this is gonna land and, and how will an IT team of the future maybe be structured? Yeah, I think we, we've got, um, you know, a, a huge learning curve, you know, particularly sort of enterprises that haven't grown up really with this new, these new models. Like one of the interesting fears I keep hearing from people, and it scares me a little bit too, is first you hear commonly people say, you know, like, you know, gen ai or these LMS are my apprentice or my assistant, right?

And then so the, like, so there's this like one thought group that says that the, the people that really know what they're doing are going to have their jobs for a long time. AI is not gonna replace them, but more and more maybe the le they're gonna need less junior people. Because, I mean, you know, let's face it.

I mean, I did something the other day that I did a year ago and GP PT three five, and, and I ran it last week on g PT four Oh. And I wrote a little sort of LinkedIn thing, and the difference was like, it, it was uncomparable the difference that of what the answer was, including code. And, and so these, the, the, the ability for these things to produce quality output, like is, you know, sort of easily equivalent to an apprentice or a junior admin or a junior developer.

So here's the, the scary part is if we don't watch what we're doing in these industries, we may dry up our supply chain of people. Mm-Hmm. Right.

So that's a, I think that's a legitimate fear that's happening, that if we keep thinking we don't need to hire as many interns, we don't need to hire as many sort of junior program or junior CI admins or junior DevOps, if you will, because we don't need as many anymore because these tools can replace 'em. Well, as these older sort of gurus roll off, what happens. So I think the answer is leadership needs to sort of understand this and pay a tax for apprenticeship free for learning for those things.

So that's one other thing. And the other thing I wanted to sort of end off, which was, I, I liked the survey data because it's creating a consistent, um, narrative, if you will, in that, like the, the surveys that you talked about, somewhere around 70% of people that are getting surveyed are saying, yes, you know, damn the torpedoes, I'm using Gen ai. And it was a, a about a month or two ago, uh, Microsoft and LinkedIn had done a really extensive like 4,000 respondees, you know, 'cause LinkedIn, you have those people available, right?

Um, and it was it very consistent, like I think theirs was like 70, uh, 70, uh, 75%, 75% are using geni at work. 4,000 people survey. And then 78% of those are saying that they're doing this, what they call 'em BYO ai bring your own ai, which is the, the shadow AI thing that I also am pretty worried about.

So, so I, I like the data of these, uh, surveys 'cause it's sort of telling the same story over and over and over. Do you think we might also look at it another way and there's gonna be a generational shift, and the newbies may be learning how to use these AI tools in college and they'll come out as quote unquote senior engineers and maybe the younger kids will take to it better than the senior folks. I, you know, again, I, if I had the crystal ball already, I'd be out on some Grady White, I'd be hanging out with Aron on his, with a boat right next to him.

But, mm-hmm. Um, the, um, but I, I think the, the real fear is it's the, um, the, the, the knowledge, the contextual knowledge of a business that gets lost in this possible, um, you know, funnel that we might be creating. So I think you're right.

I think these, these, the kids that, I mean the, the domain of how, you know, uh, you know, the, to go too far, but I, I'm really exploring this with DevOps. I think DevOps in the next five years is just going to be, or software engineering in general. But certainly what we do in DevOps, dev Stack Center three, you will just have natural language conversations with generate AI to do everything you do.

I, I'm convinced that, um, you know, that for everything that you're sort of dealing, it will know your network. It will know your, um, your sort of terraform apologies, you'll be sort of copying and pasting a terraform. You'll be, you'll literally, you know, this is this agentic processing they're talking about where we go not from just, Hey, you know, uh, cheche via copilot, write me some code or write me a Terraform.

You're gonna just say, go and fix this issue, or go ahead and build this feature. And they're, they're a bunch of open Devon, uh, uh, you know, copilot workspace from Microsoft. These tools are already showing promise here.

So I think that's gonna change. But again, back to your original question, my fear is the, the ob y Kenobi of the large banks and the sort of the retail who understand the technology and they do what they call the social technical, but the business side, I think there's a fear that we could starve that out where the, where the young kids come in and we don't, you know, again, the technology's getting solved. It's the expertise at a large company like Target to make a great business decision or business decision turns into our technology delivery where the OB one canobie of the company can say, Hey, LLM, go ahead and gimme a prototype for this solution.

As opposed to, Hey, junior programmer, you've been on job. Lemme give you a really hard, I mean, look, I know how I started. I started Exxon and the C and I was like, gimme work, gimme work.

And they gave me the most complicated problems that, you know, and I, I learned really fast because they were just booming and they couldn't do all the work they had to do. And they gave me, you know, in, you know, like writing an accounting system for a cray, um, like, you know, um, so I, that I think that would be my sort of response to your question. I kind of wanna expand on your question, Mike.

Um, I think if you look far enough down to our younger generation, um, and our kids, they're growing up more and more in this digital online world with no fear of using all these different technologies and tools. And I think, um, AI is gonna be really prevalent in their world and they're gonna take it for granted. They just use AI all the time.

Um, and they're, they're gonna have these AI skills just inherently because they're, they're online so much more and with no fear of trying out these tools. But, but again, I just, uh, right. And that, I totally agree with that.

But the big question is, will they get hired because do we really need them to solve the business problems, either Exxon or the P of a or like, you know, because the, the, the, the people who, you know, at the top of the pyramid are basically don't need them. 'cause they can use, that's, that's the, Or maybe those kids will do what the top of the pyramid's doing at a lower cost, and that'll be a factor in the equation as well. Could be.

All right, John, what does the team look like in the future? Because in my mind, it's starting to emerge. You talked about agents, and I think that each DevOps person or it person is gonna have, I don't know, seven or eight agents that are specialized for different tasks that they're trying to orchestrate.

But then I am part of a larger team and everybody on the team has their own agents. So now I got five people with, I don't know, a hundred agents that need to interact with each other. How will we kinda orchestrate all this across this a group?

Yeah. You know, for the longest time, you know, or longest time within the last two years of like studying this space and playing with the space and talking with people and doing some consulting with clients, um, I, I, I had a strong belief that, you know, the, the two pizza team model needed to include an AI expert, because I think originally I think there were a lot of companies that thought we need to hire as many AI experts as we can. But then I, you know, as we sort of seen it play out, it was like, no, we have really, it goes back to the sort of business people, you know, the two pizza team.

I have the network, I have the different componentry that I know how to deliver and build the right solutions. What I'm missing is this, you know, sort of AI knowledge, a different way to think, which again, I think is the common thread between both, you know, both of our topics, which, you know, or all the sort of like, you know, you know, future of ai, strategy of AI is, you know, companies are not sitting down and literally thinking about the discipline of massively training people in an organization. And one way to infuse training would be, uh, put, you know, put like add sort of another, uh, headcount to the two pizza team.

And, and basically, you know what, it would be a pure AI expert, you know, and you get this, and that could be a good role for a young, you know, young person outta college, right? They, they, they've gone, they've gotten their sort of MBA or, uh, you know, master's of science, uh, on AI and then throw 'em right into that, and then they get to learn. That solves my other dilemma.

But, um, but I do think the agen thing also is another, it's one thing to say we're gonna reduce coders because of co-pilot and all these tools. It's another thing when you start seeing what these army of bots can do, um, and, and be, honestly, Mike, I'm a little unclear what a team looks like. You know, if you, if you talk to some of the people working at the big fig three, big four, whatever they call 'em today, you know, you know, and it's in their best interest is they want to come in and write like a million bots for a company.

This is like RPA all over again, but in a whole nother level. And, and, um, and I, I like that scares the heck outta me. Like, you know, who's gonna do the technical debt, you know, clean up of like a hundred, you know, 10,000 bots.

Um, but, um, maybe other bots, I don't know. But, but, um, you know, agents. So I, I think the agent thing is really interesting because the, these agentic, they're the, the sort of the poster trial term for this now is agentic, um, agentic dev tools or agent DevOps or agentic.

Uh, and it really is the combination of LMS being orchestrated in such a way and personas, they can say, Hey, I'm, I'm a, I'm a system engineer. Um, I am a developer. I'm, and you literally start routing work in a very function oriented way.

So, um, so I don't know, I think it's, it's a little early in this agentic discussion to really understand what teams are gonna look like, oh, this is Great. There's probably also some, some resistance because nobody wants to think that a new technology is gonna replace them. I mean, they, they wanna use it, of course 'cause it's gonna improve productivity.

But probably there's a little bit of, of caution in terms of how much do we implement it, how much do we rely on it, you know, for, for those that work in DevOps. Right. Well, you know, Bob, Bob, I just wanted add, I'm sorry to do that, but this, this is a thread that was in those surveys as well.

Is there, it's, I thought shadow AI was gonna happen because people couldn't wait and they needed it faster. But some of the surveys are pointing out that people's fear of telling leadership that they're using ai Yeah. Might play into the we don't need you anymore.

So that Exactly. Yeah. Yeah.

Mm-Hmm. So to that point, John, who should be responsible for teaching these people and training them? Because, and it's an ongoing debate, but do I, as the employee, need to really drive this on my own?

Or should organizations have some responsibility for training their squads? 'cause a lot of times I think, um, a lot of leaders think that's on the employee to stay current. Yeah.

No, I think that's the wrong decision. I, I, I might have mentioned in one of our previous discussions, but I was at a, a large conference and, you know, one of the, um, the, the guys who, um, so it was Cigna, they, they said this publicly. So, uh, they have the guy who basically pioneered their whole cloud journey, right?

And so, like, talk about lessons learned, right? And one of the things they learned through that process is they didn't do a global education program on cloud. And so he was brought in to help this sort of new strategy goal for, you know, like they had a cloud, uh, leadership team.

Now they have an AI leadership team, right? So bring everybody in, bring the important players across hr, a different part. Um, and he's, you know, his first suggestion and they have implemented is sort of a global do's and don'ts and what you should know as sort of, um, the training.

You know, the, the big corporation training is the most non, it's the most annoying stuff you have to do, but every year you have to go through this thing and it won't let you end. It won't let you cheat. Like, like you have to answer questions, it paces you, you can't finish it quicker, um, and make it one of those.

So it actually is, and I'm not a permanent record. I feel like I'm not like a terroristic. Everything has to be, but like, I think this is one of those ones where you're sort of explained what AI means to this corporation at this time, what this corporation thinks you should be doing, and what you definitely should not be doing.

And then, and then literally once you go through that thing, it registers that you've taken that class. So you can't pull the do now ask forgiveness later. So I think global training is what, what is a common thread of all this sort of survey data of people saying, you know, they don't know they're getting started early.

They, you know, like, I, I think it, it is, it is, uh, in the best interest of a organization to put together sort of a tiger team. I wouldn't make it a per permit team, you know, because then that becomes another sort of team of teams of teams. But I, I do think there's this, um, responsible and, and really for your own sort of health as an organization to sort of figure out, you don't know everything.

It's emergent, it's all nascent, but like, at least sort of document and make everybody in the corporate, if if 78% of the people in the organization are using this technology, you got a problem. You got a real problem here. And so it, it, it really, I mean, I, I think the, you know, like state, like this is our state way.

We, we think about it right now. We don't know everything, but this is the way we're gonna operate. And I want you to check box that you agree that you're gonna operate this way.

All right folks, well, I'm gonna leave it there. But do remember that these AI engines, if you want to call 'em that, and the LLMs, they get smarter every three or four months. So staying current on this is gonna be a lifetime proposition.

I think we'll be back in a minute. com is the number one online destination for DevOps education and community building. com covers all aspects of DevOps, including DevOps, best practices and tools, DevOps culture, DevSecOps, business impact, continuous testing, continuous delivery, and more.

com has the largest collection of original DevOps content featuring breaking news, blog posts, podcasts, and more. com to learn more. com where the world meets DevOps.

All right, folks, we're back, and initially we just talked about what it means for IT folks, but the organizations as a whole seems like they're haphazardly moving down the path of ai. Nobody seems to actually have a strategic plan or how to think about this at scale, because, well, there's tons of articles on tech strong AI and elsewhere out there that are all talking about how it feels like to me. John, let's start with you there.

We're kind of just stumbling down what to do with this stuff, and do we need to take a breath and kind of think about this in a, in a longer time horizon? I, yeah, I think the biggest obstacle to this technology is, uh, you know, is that it changes. You know, David Edwards says this really well, that, you know, we've been trying to put determinism sort of, you know, truth and facts into this whole industry for what, 50 years as long as sort of like large IT data centers and stuff work, right?

And, and, and like, it's still very, it's a still very complex world, but we've, we've sort of gotten to like, we can define a supply chain and it's pretty clear and we can say what exactly happens. And we have a lot of monitoring and data. So we've, we've sort of, we've reigned in this stuff.

Now all of a sudden we have this technology that is untrainable, it is if that, I know that's not a word, but, um, it's non-deterministic. And, and what we're, we're failing to recognize is that the, these new sort, this new way of doing things is probabilistic. It isn't like one plus one will equal two.

In fact, you know, you know, early days of GPT-3 five people would go, oh, you can't use that. 'cause you can't do math. It doesn't do math.

It does approximation. This is approximate models. The answers are approximate.

They get really good. And I think the hardest part for a corporation that's been literally thinking about solving it problems including software delivery, oh, security as a controlling, you know, how much more can we control? And that is the sort of the fatal floor think it is a fatal floor thinking in these new generat AI models.

You know, people talk about hallucinations Yes. Hallucinations, approximate answers based on the data. So, um, so I think there's sort of embracing and understanding that you're dealing with a different type of world.

And, and I think that's where people come in. I, I was telling you earlier, I mean, one of the things I hear over and over is, the way you solve this problem is better data classification. No, that's not an approximate.

The, the way you solve this problem is get a better approximations of unstructured data. In fact, the, you know, we've talked about this, IDC did a report last year that 90% of the data in an, in a large organization, a commercial organization or a large organization is unstructured, and we manage 10% of it. That's the gold.

So it isn't about turning that 90% into tructure. 'cause the 90% is like spreadsheets. It's wikis, it's, and, and the goal of this thing, the gold GOLD, is that I can take that data and like turn it into knowledge in a way I could never do in the past.

But if I start thinking about like, how do I, you know, the trap of like, how do I take all those, this is what we've been trying to do for years. How do I take all those spreadsheets and wikis and turn it into tables, and then the tables can be like, that's why we've never gotten close to what we're getting with Geneve AI in these common lms now small. So again, I think there's a, like, not only is like training, but it's also like having the break, the, you know, where they break the bone to heal in terms of a way that we think about these problems differently than the way we've been thinking about solving these problems.

And I don't know if that made sense, but No, I'm, I'm with you. And to your point about deterministic versus approximations, I'm, I think, you know, there are certain things that we wanna have done the same way over and over again. But if I think about the way we humans make decisions, for the most part, we're using our experience to kind of make and guess ourselves, which isn't much different than what the machines are doing.

And who guesses better is up still up the debate in my mind, because a lot of times, uh, to be honest, you know, people are making quote unquote gut decisions, but was that gut decision, you know, indigestion from what they have for breakfast? Or was it really experience? Um, it seems like the way we humans get around that is we have a bunch of us together make decisions and gut check each other.

So won't the machines kind of do the same thing? Yeah, exactly. Exactly.

I mean, there, you know, um, not to go too deep, but there's really three forms of inference, right? And, and then sort of going back to Aristotle, right? Or, well, two aristale, but like, there's deductive reasoning, there's inductive reading and there's abductor reading.

And I know I'm going sort of crazy, but this is really important because the early and, and now what we're seeing is at least the two of them deduct inductor actually coming into play, um, like deductive is more like the symbolic expert systems we've had over the years. The, um, inductive is the sort of neural networks, the, you know, what they call subs symbolic, right? And, and the reason why, like, I'm not just throwing out big words to make me sound impressive.

The reason this is important is what we're starting to learn more and more is how does humans really, to your point, really make decisions? And it's a combination of really all three, which we really haven't explored in gen ai, which is abductive and abductive is instinct in guesses. And so this is the beauty of marrying gen AI with humans.

This is why the humans, you know, probably gonna be in the front, you know, the left, the left arrow line pilot seat for, for some time to come because humans do abductor reasoning. And to my knowledge, no gen AI implementation does abductor reasoning. And abductor reasoning is the sort of the instinct, you know, maybe I did have a great morning and somebody, and somebody when I was getting coffee was incredibly nice to me.

And I'm like, my mind is really sharp today. So when I come into that meeting, I'm, I really can actually nail problems a lot quicker than I could yesterday. Like that, that our ability, the Colombo effect is what's going to work with all these sort of agentic models.

So I think that is sort of the, if we could call it the crowdsource version of this working properly. And I think you're right, the human element will probably, and there's a lot of people believe that maybe there won't be a GI ever or a GI within a hundred years because of the humans have this other superpower, which is our instinct, our common sense. And, and you know, again, I'm not like at the, the Hinton or lacoon level to explain what they do, but I'm pretty certain none of their models have the sort of, um, common sense instinct that it'll take It.

Well, it also depends on how you interact with ai. And, and also this is a pitfall of it is the bias, the bias and the data, the humans that are providing the data and how it's, um, how it's presented. And there is different biases that can come out, like for a human, um, John's right bad, you know, bad experience at the coffee shop or whatever bad day you stub your toe, you're in a bad mood.

But with ai, um, there can be that bias. I think that's something, it was talking about crowdsourcing, but maybe having, um, what I find useful is to use different AI models and see, well, what does this one say? What does that one say?

Because to get the consensus, and that might be something that we do going forward. That that's, Yeah. Sorry, go ahead, Amanda.

And I, I think it also comes down to, again, companies do need to understand exactly how these AI tools work and their limitation, and that they are just augmentation tools. And then they need to have a clear understanding of what problems do they have and what potential solutions could the AI tools, um, provide. And then a clear use case and, and end goal moving forward, and then properly tracking it through each step to ensure that it is working as they hoped.

You know, bias is not a bad thing. And here's why I would say, um, we all have biases with everybody we meet. And I'm not talking about race, creed of color.

I'm just talking about your experience with that person and how much you trust them based on their expertise. And you factor that into, if I tell you something or somebody else tells you something, you weigh that. Yes.

And I think we need the same thing when it comes to these AI models. We need a little more transparency into how they function and what their level of accuracy is so we can apply that bias. And so, John, do we need more transparency into these AI models?

There's gonna be a lot of them. We're gonna mix and match 'em as we go along, but I think, do we trust them too much? Yeah, no, I mean, and this is a problem, right?

Because Bonnie's right, and you're both right. I mean, this bias doesn't really have a negative or a positive or con connotation. It's just bias, you know?

Um, you know, the, the, the coffee, you know, the scenario is, is a bias, right? I mean, I think that the biggest thing that people would say about ai, these sort of the anti a GI people, which I'm, I, you know, actually fall in, fell into this, uh, this trap of, now somebody sucked me in and I'm doing a ton of research that I didn't have a whole lot of time for, but is, um, the, the, the models can can't learn like humans, you know, that point of the coffee, the, the, the model can't be terrible on a Monday and great on a Tuesday, because on Monday, the person who sold me coffee was just miserably, you know? And the person on Tuesday was just delightful person who brightens up the rest of your day.

And like, that's just a small of millions and maybe trillions of transactions you even have minute by minute. And so they, they call it the sort of monotonic learning that a model has. So, and then the thing about the bias, I have a kind of funny story.

I was given a presentation and I wanted to make a picture of myself through Lily. I just stuck with chat g pt, I didn't even use n the other. And, and I asked, I said, here's botch glue, here's my bio, this is what I've done.

Can you do it like this? Can you create a nice bookshelf like I'm old? And, and make it older?

And it made a really good looking old guy. It was like perfect hair. And I said, well, I said, how about somebody who's more bald and, and a a little, and the face is a little fatter?

And it came back, it still made a really good looking bald. And, and I finally had to say to it, and this, there's a whole book written, uh, unmasking AI by Joy Lummi. So it's a brilliant book about like bias.

And again, there are good forms of bias and bad forms of bias. The good bias is that it can write novels. For us, the bad time of bias is it literally could have a police force arrest somebody based on a Geneve AI profile, right?

But like in this case, finally I had to say, okay, I want close to bald, um, older, fatter face and not as good looking. And it literally still, it got, got reasonably close. But the problem was the training model for almost every image of a human.

I, I think in Enjoy Blue's book, she talks about, like, I, I'm just gonna make the numbers up, but they're incredibly high. Like, you know, 70, 80% of the trained data is good looking white males. But so think about that across all spectrums, not just images, but texts, all the texts that's been written.

So anyway, so there, but, but the, the key answer to your question, Mike, is that sort of the field that everybody's sort of screaming and hollering about, and sort of, I might too, to, to, to-do list of a long list of things I don't have time for, but explainability and, and I think that's a topic that, uh, is really worth trying to understand, trying to get the, and, and the model providers are, are notoriously bad, you know, not only even some of the open source ones don't provide their training weights, right? So there's a whole difference between completely closed. We don't even know what they do.

We don't know what they do. There's, oh, we're open source, but we don't provide the training weights, which basically good thanks, you know, like, um, you know, thanks, but no thanks. So, Well, let me ask you another question about that, because, um, I'm trying to figure out why are these AI models so quote unquote overly helpful to the point where, um, they will tell me something that they think I want to hear, or they're just trying to provide some sort of answer rather than just saying, I don't know.

Or probably I just, here's my best guess. But it feels like, uh, we're waiting these things to be my best friend when I just needed to be quote unquote helpful. Mm-Hmm.

Yeah, I mean that, I mean, that's part of sort of prop engineering, right? Like almost every canonical 1 0 1 props is, if you don't know the answer, don't, don't answer or say, I don't know. Right?

So that's sort of a 1 0 1 prop. So, so, yeah, I mean that, you know, I mean, but you have the same problem with humans, right? Let's say there are, there's always the one in the crowd that, you know, that cheers the guy cheers, right?

Uh, nor not Norm, but the other guy, right? That would cliff cl he would answer every question. Mm-Hmm.

You always have one of those people that you work with. So the thing I always like to say is people hallucinate as well, you know, computers, you know, lms, um, people give you wrong answer. The cliff klavin of the company, if you ask 'em, Hey, you know, I got this problem, and you know, there's this library, they're like, it's just an instinct to tell you a wrong answer, or they'll tell you an answer they think is right, but they don't really know the answer based on their level of knowledge.

They're giving you the right answer, but it's wrong. So I'm like, okay, cliff, have you ever worked with this library? No, I actually haven't.

Okay, now I'm gonna go to somebody else. Right? Like, that's the kind of dialogue you have to have with these LLMs, you know, you, you, you have to think about, because the key is the natural language.

And the thing we have to learn is how do we have discussions with humans to solve problems? And even though we're not dealing with human, we need to sort of apply those type of, and, and to your point, Mike, you know, the, the fatal floor is that people think they're magic and like, oh, I'm just gonna check their p and gimme the answer. Like, but you, most people are intelligent enough to not go to Cliff and say, Hey, cliff, you know, gimme the answer.

Gimme the stock that I should buy today. I, you know, Mm-Hmm. Yeah.

Like, we don't do that with people. All right? I guess folks, whether you're taking advice from a machine or a person, you should consider the source.

'cause basically, you know, the information is only as good as what went into it, folks, we'll be back with our third block in a minute. Hi everyone, and welcome back to the Techron gang. Well Meta, formerly known as Facebook has a new version of their llama AI model.

This is open source, and it's being noted for its energy efficiency and scalability. 1. Hi, I am Bonnie Schneider with your Ecotech analyst Insights Mets.

1 marks a notable shift on the AI landscape. 1 employs quantization a technique that streamlines calculations using lower precision numbers. 1 competes well with top models across various tasks.

However, these results await real world validation. The model is flexible, available in multiple sizes. Organizations can select the version best suited for their needs and resources.

This scalability could drive energy savings as AI adoption expands. Meta, CEO Mark Zuckerberg highlights llama's open source nature. He says he wants to ensure that more people around the world have access to the benefits and opportunities of ai, so that power isn't concentrated in the hands of a small number of companies.

1 demands significant computing power for training. This could potentially limit adoption by smaller companies. The open source model also raises valid concerns about misuse without proper safeguards.

1 could play a crucial role in driving a broader transition towards open source energy efficient solutions. 1 is open source, and as you saw, mark Zuckerberg is very optimistic about this way going forward. 1, what makes it unique and energy efficient and scalable.

And, uh, John, I know you have some thoughts on this process that we were just talking about. Yeah, you know, I, I guess I'll start with, uh, who would've sunk that? Zuckerberg, we were playing the role of David in the David Goliath story, but, um, yeah, no, the, the ization is really interesting.

I went in, you know, I, I sort of watched your video. I've been trying to bring, uh, three one up. I'm really, really excited on it.

Like, I think you heard in past shows we've had why, you know, I think I, I really want the dark horse of open source to, to win because, um, I think it's gonna keep the industry honest. I think there's so many positives here, but yeah, the colonization is, you know, again, you know, you know, Facebook, their technology and their innovation technology is, is always amazing. And, uh, they, you know, the, this idea that they sort of reduce the, uh, so the, um, the processing footprint, you know, and two technical, they're going so 32 bit floating points to like eight bit, um, scale that sort of changes the cost, the power, you know, the aggregation of, of just changing the, um, the, the processing, um, you know, you know, sort of word length as they call it in sort of this old assembler programs.

But it, it becomes this interesting trade off and like only a scale like, uh, sort of medic could probably pull this off. Like, like there's a little bit of hit on performance, but the cost outweighs, you know, the aggregate cost. Uh, and then, you know, the, um, the, the, the bigger thing is it becomes much cheaper for intervals.

So it's a little more costly. I mean more, more processing, uh, time for the training. But again, most people using these models are doing more inference than training.

You know, my survey is, you know, informal surveys that target people. Very few large enterprises are training models in this new gen ag space. And, um, so the, the inference savings, I think is another great part of the trade off.

And then, you know, it, it allows wire usage. 'cause you, you don't have to have really sort of very complex GPU rigs. At least that's in theory.

I, I haven't prototype or tried this. So, yeah, I think that, um, a really, the ization is I think a sort of, not a great way to describe it. 'cause your mind goes right to quantum computing and it is not quantum computing.

So maybe they wanted that effect, but, but if you sort of look at it, it's really sort of reducing the sort of instruction word length, which then has like vast implications of cost savings and u utilization, all the things that add up to running large data centers. That's true. You know, I'll, there are different analogies, uh, to describe it.

One of them was using, taking a picture and having one at different resolutions, but getting the same results. So, or have, yeah. So I think there, there's different ways to kind of break it down, but, um, one of the things that's interesting about this model, I mentioned that it's scalable.

So different businesses, let's say, um, uh, we're looking at small, the John or smaller ones, they might be able to get the smaller version of this model or the medium sized version of this model. So it does have that scalability. Yeah, no, and, and then, you know, again, I think what's also fascinating, they, they're comparing, you know, um, the, uh, you know, the 4 0 5 B, right?

Which is the, you know, 405 billion parameter set is still well under GPTs, but they're actually saying that, um, it really is, you know, 16, it took 'em 16 months to catch up to what is GPT-4, which again, is another great sign for the dark course of open source. Um, so, you know, and, and if you start looking at sort of what it claims to do or what, you know, Jan Laun who works at matters as part of this group, has published some interesting sort of comparisons on LinkedIn. Um, they're saying that we have an open source model that in, in many respects competes directly with, um, you know, GBD four and maybe G PT four.

Oh, I, I, I'll be cautious there. gbd four, let's not go crazy. So that's significant.

So John, will open source ultimately carry the day here in the way it has in other categories? Is it gonna play out the same way? Because, um, right now proprietary models seem to be, um, have the edge in terms of performance and capabilities, but over the time will open source kind and carry it Well, I mean, it just to be a little contrarian, like open source still hasn't carried the, the, the day in like other fields, right?

It, it is like enormously being successful in the aggregate of everything that's happened. You know, the, you couldn't have Amazon without open source. You, you probably couldn't, you couldn't have Uber, you couldn't have had Airbnb, like all these sort of, but I don't, look, we still struggle, you know, like when I, I sort of, I do a lot of valuations for startups, right?

And, and like it is, it's a coin flip and probably even more than a coin flip, uh, uh, that for in the, the going proprietary route than it is open source. 'cause the business model still doesn't seem, you know, there's only been, in my mind, one successful company that has really taken advantage of open source and that's Red Hat, right? So, um, and answer your question, uh, the, um, I, I think we need it.

I've said this over and over in this, you know, in this text on gang, uh, interviews that we've done and, and is that we, it, it, like, it has, it has to keep, we have to keep the balance. 'cause we need to transparency, we need the explainability, we need the ability for, um, you know, for, you know, what we don't want is three companies owning this, right? And that's what Zuckerberg is saying.

Um, uh, but, but you know, I think, think how this all plays out is still because, I mean, you know, the, I don't know what when the GT five is coming, but like the difference between GPT-4 and four, oh it was in my experience, incredible. So like if they're, if they've got bank, like, so okay, we caught up and then here's the other like little bit of scary part I have is if Met is the only one doing this, then what does that mean? Right?

I mean, is sru and they, they got great funding, but no Met is right now, you know, may maybe, depending on what Nvidia wants to do, I don't know, maybe I'm sort of going crazy here, but, but like, if it's only one player that can even keep up, and if like, we don't, like, do we really trust Facebook to always do the right thing here, you know? So, uh, so I don't know. It, it is, again, the only thing I can do is like what I've been doing pretty much the last, the second half of my career is rooting for open source and hope in, in believing that open source is a, a great vector for just many levels of this, not just monetarily.

Mm-Hmm. I have another question kind of unrelated, but you mentioned small models. So I'm, we keep using the phrase large language models, but you hear the phrase small language models now and medium sized language models, and they seem to be coming in T-shirt sizes, Small and large.

Mm-Hmm. Um, So do we need to kind of change the way we think about language models? Because the terminology is, we've moved beyond it.

I mean, I guess I'll go first. I mean, I think we talked about this in a previous, but the, the, the way the small model language models are evolving is very fit for purpose, right? Like if I'm doing edge work and I have a very specific domain that I wanna solve problems for, you know, good friend of mine, Justin Enox, he works at a company division.

Uh, he, he was at our, um, on our first gen hackathon, right? Um, EVT and you know, he's doing work for, uh, California Edison, right? And they're building what, when they started, they weren't calling small language war, but he is literally waiting these LLMs for people to climb up on telephone poles who need to get real accurate information for how to fix those incredibly dangerous cons things, right?

And so that doesn't need to have a model that can tell you who Aristotle is or what objective reasoning is, or right. It just needs to get you to the system of record answer that is legally bound that basically doesn't kill the technician. Um, you know, so, um, so I think that's where these small language models come in.

Now, they're smaller, they're efficient, they can sort of be very prevalent at the edge, and they're, they, they're not designed to be general purpose. I'll answer any question you need. They're designed to be, and they've been trained to be very domain specific to solve very specific problems.

So Right. Whether we keep calling 'em small, medium, large Or, or mini, that's what that, that's what that chat Jamie t it has. Now maybe We can call them more bespoke models.

I think that that's Great way. But, so bringing the two previous thoughts together, is it possible that we might see open source models that are smaller, become more prevalent to the larger models, become more proprietary, uh, because they just take so much more effort to build and, and will there be some sort of split along size? Yeah, I mean, the, the truth of the matter is, I think that we over rotate on this discussion about training.

1, which, oh, it's gonna be better for training. I mean, um, if you think about what people were doing before Gen ai, they were training a lot of models, but they were probably in the grand scheme of things, small amounts models, if you were doing Monte Carlo simulations or financial sort of models, you know, machine learning, you know, pre gen ai right? Was very specific in an organization to solve very specific problems, you know, ad revenue or, um, you know, this financial modeling or those things, right?

And gen AI came along creating these insane models that can answer anything. It's scraped 60% of the internet. It's got all this stuff, right?

So really what the small language models are is just this sort of new generation of technology, you know, the transformer model to allow you to kind of go back and do more with less expertise. I mean, that's probably the biggest story. Like if you were creating, um, these very, you know, predictive financial models using machine learning in like a large bank.

You had a small group of very, very high paid specialists. 1 has an 8 billion, uh, parameter, right? Like an eight B, and I, maybe I can train it, but, but again, I think less training is going on in the organization.

'cause that's hard. The, the, the more likely is taking p and building like a augmentation generation or build your own vector databases, which is a lot easier. A lot more people can participate in that example.

And you can get high, you know, high efficacy answers, you know, in some cases maybe as, as good is, uh, the ones that would be done by this sort of machine learning experts. I think it also, again, comes down to each company and their needs and, and, and depending on their needs would depend on the model also their resources. Okay.

Hey, you were, you started this conversation talking about sustainability and mm-Hmm. I don't know that they, we've talked about sustainability in the past, but is anybody ranking AI models based on their sustainability factor? I think so.

I think that's, that's a great point because I, it is becoming a little more competitive. If you watch all the recent releases from, um, AI just recently chat, GBT has chat GBT mini, and one of the, I guess, selling points of it is it uses less energy. So yes, the pressure is on, i i, we talked about a few days ago, water resources being, uh, overwhelmed by ai.

So how are we going to offset the energy in, in, in various ways is absolutely, uh, coming into the conversation. 1, that this is a more energy efficient model. This is what they led with.

All right folks, I think that's where we are for the day. We're gonna come to an end on what I guess I will refer to now as AI Monday. Thanks for spending some time with us.

We much appreciate it. And, um, please stay tuned. The rest of the Techstrong TV lineup is following shortly and it's just gonna be another awesome day of outstanding programming, and we appreciate all of you who spend some time with us and we look forward to talking to you again tomorrow.

I'm Bonnie Schneider, sustainability contributor to the Techstrong Group. I'm excited to introduce you to a groundbreaking new initiative from Techstrong Research, the sustainability pulse meter. The pulse meter offers valuable insights into how environmental responsibility factors into tech purchasing decisions for key players in the industry.

Position your company as a leader in the industry and differentiate from your competitors with a sustainability pulse meter offered exclusively from Techstrong research.