Techstrong Gang June 11, 2024

Hey, everyone. Happy Tuesday to you. I'm happy to be back here down in Florida at our Boca Ratone Studios.

I'm Alan Shiel, and as we say, some of my Caribbean friends here in Florida. We've got some distance, some debt for you on the gang today. You're watching Text and Gang.

Hi everyone. Welcome to Texture Gang here. It's Tuesday.

We've got a a, a good potpourri of, of different stories we wanna cover you with. Today we're gonna talk a little bit about Agile. Is it all cracked up to be or is it a ha Well, we'll talk about it.

Uh, we're gonna talk about what it takes to be a modern CIO today in the age of ai. And lastly, we're gonna talk about a study coming outta Stanford about, uh, RAG and what may be in the future for you with Copilots. But before we jump into that, let me introduce you to our gang members for today.

First of all, he is out in Philadelphia attending, I guess it's the, what is it? AWS Not Reinvent, reinforce. It's our, yep.

Our CTO and Principal Security Research Analyst. Mitch Ashley. Hey Mitchell.

Good to be here. Good to be here wherever I am. It's good to be here In Philly.

And then joining us back from our summer hiatus from San Angelo, Texas, it's Amanda Ani. Our Hello everyone. Hey, Amanda.

Good to have you on. And then our Special Gang member today, he's been on here before, though. He is, I'm not sure he is wearing his colors though.

It's our friend Steven FoST. Steven of course, is the CEO and founder of Tech Field Day, and he's fresh off of Tech Field Day. Hopefully Steven, didn't he?

I know he was Cisco Live, but wasn't there another one? Yeah, we were, we were at both Cisco Live and at, uh, click Connect last week. And we've got Cloud Field Day this week, and we had App Dev Field Day the week before, so it's been pretty busy.

Um, but yeah, we, I, we, we divided and conquered. So I went to, uh, click Connect and learned a heck of a lot about data and AI while Tom Hollingsworth was over at, uh, at Cisco live, uh, deep diving into, uh, networking and AI because it's 2024 and everything's about ai. Absolutely.

Speaking about ai, I can tell you that this woman is real, but she knows something about ai. It's our friend from Mire, Tracy Madden. Hey, Tracy.

Welcome. Hi. I am looking forward to all the topics you guys have been, uh, suggesting for today.

This is gonna be fun. Fun. I hope so.

That's what we try to do, and we hope you enjoy it out there. The last person I want you introdu in, not the last person I want to introduce, but the last person I will introduce or should, or should introduce is my co-host here in our Boker headquarters. It's our Chief Content Officer, the one and only Mike Ard.

Hey, Mike, how are you? Good to have you back. Absolutely.

It's good to be back. I've been on the road and act. I leave again tomorrow, actually.

But such as life, as Steven was saying, you know, it's, it's that time of year. Um, so guys, let, let's kick it off. We got this.

What's wrong with Agile? Mike? I, I've got some views on this, but I'll let you do the intro.

All right. Well, there's another study, and this is not the first time we've been talking about this issue. It's been around for a long time, but the study says that folks who are building, uh, agile applications are somewhere in the neighborhood of 268% more likely to fail.

And this conversation's been going on forever. And yet we still have people who are diehard believers and must be getting some value out of it. So let's start with Tracy, though.

You've kind of been around this topic for a long time. What's your thoughts about this study and, and what are the issues with Agile and is it just part of the thing, or is there something fundamentally wrong? Oh, gosh, there, there's a lot to unpack there.

The study's interesting. I, I read through it. And the small overall, a small group of participants, 350, but that's not insignificant.

Is agile. All it's, um, meant to be. Is it all It's talked up to be?

Well, it's like everything else. It depends. A lot of people come at this with a, a zealotry, you know, they read something, they have this high end, high mindset of what it needs to be, and they forget about all the basics.

And I think a lot of what we see is when people become, uh, focused, wrapped around the axle of what they think it is, instead of tailoring it to what they need. I think it's where you see a lot of the, the issues. There's also resistance, tons of resistance that we see.

Does everybody fail with agile? No. Does it belong everywhere in, uh, in a purist sense?

No. It always needs to be really tailored to what you're trying to accomplish, the people that you have, the culture of your organization, your unique context, Thoughts. I know you're dying to jump In.

Yeah, no, I've been, that's why I wanted you to introduce it this way. I don't have to be partial or impartial. So I, I've gotta call bull crap on this chop job by the guy at the register.

Some, they're even publishing this on there. This is a study by a company whose premise is, if you don't put all of your, all of your plans beforehand, all laid out, you're bound to fail. And so they latch onto the one of the manifesto tenements, which is good software over good planning, in essence, is, is the thing to do.

And that is key to Agile. But the the problem is, is as usual, they, they made it as if, yeah, if you don't have any planning at all, you are 264% more likely to fail. That's great.

No one does no planning at all. The, the part of this agile thing about, uh, you know, the tenant about good software over good planning. It, it doesn't mean you don't plan, it just means that you've gotta adapt to the time and circumstances and what, and your environment and what's going on chop Job.

But that's misunderstanding Fake news. Right? But that's the thing that people don't understand, right?

Let's just go, we're gonna do, we're gonna do this agile thing. I I wrote something about a year and a half ago, almost two years ago, because somebody said to me, we don't need any architects. We don't need any engineers.

We're doing agile. And my brain exploded. Because you do need to make choices.

There is planning upfront, of course. It means, means we don't plan everything upfront. So I think that agile, for a long time, for a very long time, has been bastardized, at least the, the, the methodology, the written out things that we are supposed to do.

It's all supposed to be suggestions on how you should behave, should you have a conversation, instead of having those big documents that we would create and we would redline them and it would take months to get after them. And then you'd send me the red line document and I'd study that gone. Those need to go away in that, because I can come to you and ask, what, what would you, what are you looking for?

What do you think about these things? So there was some goodness there that got totally misunderstood. And yeah, I, I would agree with you that this study, um, paints it too, too bleak.

Too bleak. But we have been having issues for a while. Well, they Haven't, We've been having all kinds of issues for a while.

'cause nothing's perfect and you gotta keep trying. But Agile's been around 25 plus years. Mm-Hmm.

It's not the new kid on the block anymore. Somebody's using it for something after 25 years. Mm-Hmm.

All right. I, I gotta jump in here too. You, you're telling me, Hey, it's, uh, 2020, uh, 2001 and we need our agile back.

Is that where we're at? It took it this long to Oh, I like That. No, we need our waterfall back.

That's what they're trying to say. It's 21. Where'd that waterfall go?

All Let's get there. One of the thing, oh, go ahead. No, Amanda, go ahead.

Talk. As I'm listening to this, and I'm thinking that some of the things that I hear from business leaders and staff and companies is it's more about the over planning during the agile process. There's too many managers managing everything and too many meetings, and it's actually becoming less agile the more they try to be agile.

Well, there's, there's another thing here too. And I, I'm gonna say, I'm not gonna say that the register guy was wrong. I'm gonna say that that is a prevailing idea in, in the world.

Everybody wants to use Agile as a punching bag. Everybody wants to say Agile is just a bunch of hackers with no planning and no methodology. And, and maybe some people are like that.

Certainly I have met people who claim to be agile developers, when in reality they're just lone wolf, I'm gonna do what I'm gonna do because I know best. And you don't know nothing. You project managers.

And, and the, that's not agile. Agile is a methodology. It has principles, it has values.

The whole point of agile was supposed to be that you're breaking down, you're breaking free from this problematic waterfall mentality. And instead you are approaching tasks in a realistic way and saying, there are, there are little achievements. We need to have little goals.

We need to have the ability to adapt. I mean, for me, the biggest thing is that you need to welcome change. And you need to welcome and embrace the reality as the project evolves and as the development happens.

And unfortunately, what's happened over the ti over time is Agile has become the punching bag that everybody loves to hate because it's the, it's the symbol to them of what they hate. It's like politics. Everybody loves to hate the other guy.

And Agile's the other guy. It's just the religious wars of, uh, methodologies kept at the same time as the clone wars, by the way. So, Well, I think if we go, if we bring it back to basics on this, it's always been to your point about the, the principles and the values that are there.

And on pausing, pausing every so often and looking at that, are we aligning to this? And that's what principles are supposed to be. Those are the things that you, if you're making a decision that are contrary to those principles, you got a problem.

But there's also that misunderstanding. And that I, I like where you went with this, that punching bag of, if something goes wrong, what do I wanna blame it on? Well, I'll blame it on that new new.

And it is still new. It's still new in a lot of places. It's still new.

They blame it on that. Oh, it was, it was because we did this. It was because of that.

DevOps killed us. Agile killed us. No, no, no, no, no.

What happened was that you didn't take a step back plan, extra time for allowing yourselves to have new workflows, plan extra time for allowing people to adjust to these new ways of doing things. And yes, I say new, but you know, it is there. The principles and the values are the core part of it.

I think there are also organizations that dive in immediately and say, we need the most hardened methodology possible. We're going to use safe, or we're going to use one of the other very over documented, uh, approaches or overly documented approaches. And they also get wrapped around the axle of having everything written down and having everything So prefigured, pre-configured preser, that they lose the agile mindset.

They lose that ability to say, you know, this isn't working so well. We need to pull back that. You know, that one pager that we said, that was how we were going to do things.

We need to make some changes there. So they, everything is still getting written in, uh, concrete, written in stone. You, you Bring out something really important.

Trace, which is a lot of cultures, business cultures, I'm just talking about the organization, not even within it, are not compatible with doing an agile approach. If you're highly scripted, highly measured, highly planned, disciplined, et cetera, this is what has to be laid out in that kind of culture. A product manager not knowing when the product's gonna be released, which is not gonna be that date anyway, um, you know, it is begging for a date from the software team, cross-functional or whatever, and they get frustrated, right?

And so there are organizations where it just won't work 'cause it's not compatible As much as you wanna try and plan and adapt and change other places, it works great. People are very much bought into it and we'll make it work. So I think it says more about the organization than necessarily just about the people or the process.

You know what, you're Right though guys, people should remember. They call it agile. And if you don't know what agile means, go look up the dictionary term agile.

It means you're agile, you're nimble, you're able to adapt and evolve and, and respond. Hence the word agile. But, you know, but this also Steven and Chasey, you, you hit on it.

I'll, I'll piggyback on that. It's the same issue, this punching bag thing going on with DevOps now. 'cause if you think agile wasn't wrapped down tight enough, their heads explode over DevOps, right?

Mm-Hmm. Because we don't have a manifest though, and we don't have, you know, the, that whole kind of on purpose, by the way, have you ever, do you ever wanna have a great conversation, speak to John Willis or, or, or Patrick Dubois or Damon Edwards or any of the, you know, the DevOps folks got started. They purposely stayed away from doing the manifestos Mm-Hmm.

And the dogma of, of having it so structured out like that. And for people who can't wrap their heads around that this is, you know, this is nuclear bomb worthy there, that literally their heads explode. And, and we're seeing that.

And that's part of running back to platform engineering because we'll know what to expect. And I'm not, not your platform engineering. I'm a fan as Mitchell.

I know you are too. And I'll tell you, we're gonna be doing something here at Techron around platform engineering. We'll announce more in the coming weeks.

But, um, it's, it's far from a failure. And when I see clickbait kinda articles in headlines like this, my head explodes. Anyway, wait, wait.

One last thought. One, I'm gonna give my last word here So I scratch my head. Because if you need a 40 page manifesto to understand that failing to plan is planning to fail, God bless you.

But I think, you know, great minds have kind of covered this issue before and it just applies to software. Okay? And that's the final word.

We're gonna take a break on the gang here. We'll be back in a minute. com is the number one online destination for DevOps education and community building.

com covers all aspects of DevOps, including DevOps, best practices and tools, DevOps culture, DevSecOps, business impact, continuous testing, continuous delivery and more. com has the largest collection of original DevOps content featuring breaking news, blog posts, podcasts, and more. com to learn more.

com where the world meets DevOps. All right, welcome back. After that spirited conversation about agile, we decided to take on yet another, well shall we say, controversial subject these days.

But there's a new study out from Deloitte talking about what it takes to be a modern CIO in this digital transformation age. Amanda, this is on digital cxo, let's start with you. But what is your take about this conversation?

'cause I feel like, um, we've been trying to figure out what the perfect CIO should be like for decades now and, you know, is it different in your mind or is this just kind of the same conversation dressed up in digital transformation talk? Well, I think, you know, this article is talking about CIOs, but it relates to a lot of positions in the fact that they can't just be in one specific niche and, and be really good at technology. They need to be good communicators, they need to be good project managers.

Um, they need to be, uh, understanding the business from the inside out, which is kind of helpful for many roles. Not just, not just this particular role. So it's just saying, well-rounded and not just being good at technology, that that's not enough.

All right, Steven, you have a lot of gray hair and I know that means that you've been listening to this conversation for a long, long time. And I can't remember the first time we talked about this. It might have been, I was at computer world, I think talking about the divide between IT and the business.

And we're still having this conversation. So why are we not making enough progress? Or maybe we are and we just don't realize it.

Well, thanks a lot first for calling me old. I really appreciate that, Mike. Um, but I'll say has hair.

Yes, I got a lot of gray hairs here. Um, I've been in it my whole career and, um, and it has always been very obvious to me what this, uh, survey is saying. And I actually agree with this survey that a good CIO is not about technology.

A good CIO is about the business. And CIOs should be focused as this survey shows on delivering value to the business, serving as change agents and basically translating between the technical people and the business people. And from my perspective, in the companies that I've worked for that have had very, like, like good CIOs from a company perspective, those people have come from outside of technology.

Many of the best CIOs I know were formerly CFOs and CROs and, and and things like that. And, and, and that makes a people like me in it kind of cringe because we're like, oh, great, here comes Mr. Sport jacket to tell me which, uh, you know, system I should deploy.

But when I gave them a chance and when I listened to them, and when I saw that as a way to get the objectives of it done, it made a lot of sense. And I think that that's really what I'm kind of reading into. Maybe I'm just bringing my own perspective into this, but I would say that for the techies out there, like me, look at that Mr.

Sports jacket from finance and, and, and, and, and realize that his job is to help you get your job done in an effective way. Because what holds back it again and again and again, is that we try to do technology in a vacuum and we don't focus on what the business wants. I'm gonna add onto that.

There is an imperative for that CIO to be tech savvy. Where the failures happen is when the CIO is tech ignorant when they, they are not investing and understanding, right? And that's where the technologists have an imperative to make sure that you're having the conversation, you're feeding them the, the articles at the right level, right?

You're having in a conversation at the right level to make sure not just your CIO but the entire organization is tech savvy. But that CIO, he's gonna partner. He's gotta have at least some ideas on what the art of the possible is.

Does not need to be a deep diver, but he has to have, he she needs some measure of, of tech savviness. Absolutely. Mitchell.

Mitchell, I want you to jump in here. 'cause apparently I have to talk really slowly for you technical people to understand. But I think what they're saying is that you are unable to, uh, communicate your ideas.

And yet I work with you every day and I seem to understand what you're trying to tell me most of the time. Well, let me write some Python code to interpret what you're telling me, Mike. Um, you know, having been a CIO at a few companies, I can tell you it's no fun to report to the CFO or to the COO in a company that does not value tech.

It sucks. And it sometimes it happens that way because, um, technology is not really understood what its role should be or could be in the business or it's runaway costs. And so it's get to get things under control.

And, and that has happened a lot in it. But I think we've seen a shift technology wise in general from being a back office function to the businesses about technology. How many times have we heard that, um, you know, we are a technology company or we, we are a software company, we are an AI company, whatever it is.

What that means is every part of the business leverages and is dependent upon delivering this. And I think the role of the CIO and I can tell you having to report to the senior level or being part of the executive team, your role there is to paint a vision as well as, as, uh, connect the dots between the strategy, the business plans, where we want to go, the changes that are happening and are, are in effect today. But what other contingencies do we need to plan for?

And then you've gotta kind steer this ship of what's happening not only in IT, but throughout the entire organization. 'cause guess what? Everybody thinks they can run it.

That's just kind of, 'cause we all work in technology and use it in our everyday lives. And every executive now plays a part of their role is to be a tech executive. Doesn't mean they're tech savvy on everything.

Maybe not on a lot, but their job is in the boardroom and in the executive suite is how do we leverage technology to our advantage? 'cause our competitors' doing it. And if we don't do it, we're gonna get our butts kicked in market.

So if I don't understand it, I'm expecting my CIO or my CTO or my CSO to help make sure we understand we've got the right strategy in place that fits with our business, where we're going and the right investment and take it from there. Other than that, I don't have a lot to say about this topic. Well, you do have a lot to say there.

That's really awesome. It's no longer a cost center though, isn't that a part of the mindset shift? So we have to take it from being it as a cost center.

It's a, it's a nasty that we have to do. It's like paying somebody run cables in the bill. No, it's, it's more than that.

It's no longer, um, just something that's serving us internally. It's no longer just the people making sure that you have a laptop or you have a desktop that helps you get your things done. It's a critical part of the entire strategy of organizations.

Now. There's no organization that's not touched in some way by technology. It's got to be elevated as a, as a key strategic partner at the table.

It just can't become a thing to itself. I think that's mm-hmm. No, it can, but I, I think you guys didn't quite finish the finish, cross the finish line there.

It has, in many organizations, it is moved from a cost center to a profit center. It's more than just helping the business to run, it's turning a profit, right? And whether when every company's a software company and we rely on our apps to carry the business forward, it's a profit center.

But you know, this highlights, it's a similar conundrum for CSOs as well, as well as other C levels. As an end said, these C levels, for the most part stand as a, a Nexus point, a n, a nexus point between talking business to the business itself, right? At the c-level, board of directors and through the organization.

And then translating that business speak in the case of the CIO to it speak so that his team understands it in relations to their role and how their roles figure into the larger business goals. Um, and I think the best CIOs do that, right? Steven, here's a great Example.

Are you optimistic? Yeah. Steven, are you optimistic about on this?

Hold on, Mitch. Mitch, go ahead. Oh, I've just, you know, I'm in Philadelphia and so I kinda have to bring up the example of Comcast and Tony Warner and Comcast.

I think you were gonna say Ben Franklin. Well, yeah, that too. Electricity important stuff.

No, I mean, using Comcast as an example, they went from a market where they're entirely dependent upon third parties to create set top boxes and operating systems and program guides and all the cable technology that we used used to watch on, you remember those really crappy program guides we used to have, well now, I mean basically they took the matters into their own hands. They created their own operating system, their own set, top box technology. They became a technology company to the point where they have Comcast technology solutions where they spin this off.

They now sell set top boxes. The Xfinity that we use if you're a Comcast customer, but if you're part of Shaw or, uh, any number of other cable companies, they've licensed that and they sell it to other companies. So they took technology, the bull by the horns and said, we're not gonna be beholden to everybody else.

Even make their own, uh, OEM equipment for, um, you know, security systems in your home that you buy through, through, uh, the Xfinity service. So that's a great example where it wasn't just the CIO, it took the CIO, the CTO and the leadership of the organization to say, this is important for us to be able to compete in the future, and we're gonna make ourselves not just invest, we're gonna make ourselves extremely good at this and competitive and market. So it, it's a great example of what you can do to, to leverage technology as your business, but not just as the CIO or one in the executive suite.

All right, Steven, let's come to you because in your travels, and we are on many of the same circuits, but it seems to me, I don't run into that many IT people anymore who are completely ignorant about the business. And a lot of the business folks I meet are much savvier about it. So are we making some progress here?

What's your sense of where we are? I definitely do think we're making some progress here. Uh, you know, as, uh, you know, we, we just heard, uh, with Comcast, I think probably the ultimate example of this is, uh, Amazon, uh, Andy Jassy, uh, sees the opportunity to make Amazon into the most fundamental tech company in the modern world.

And that was not a foregone conclusion. Look at some of their competitors. It it, it didn't happen by accident.

And I think that that shows the opportunity. And like you say, I'm seeing more and more people in the IT space who are getting savvy to this. Many of them because they've spent, you know, they spent years earlier in their career, you getting burned working on projects that weren't connected to the business.

And they are seeing the opportunity to translate that from IT, speak to business speak, and they see that as a career growth opportunity. It absolutely is. That was how, you know, that was my career.

That's what I did. Also, I think that on the other side, the business is getting more technically savvy because all of society is getting more technically savvy. Um, you know, I was at, uh, this click connect event and uh, one of the speakers, an independent third party, you know, outsider, um, said you can be good at, at, at, I'm sorry, you can be good at AI or bad at business.

AI is a meta technology. And I was very struck by that comment enough that I wrote it down because I thought, wow, is this the world that we're living in where technology is now fundamental to all of business and AI technology is fundamental to all of business, and therefore everyone from the non-tech side, from the tech side, we all owe it to ourselves to come together and figure out how to use this meta technology to make our business as good at business in the future. I, I think that, that it's, it's, it's converging.

I I don't disagree with that. Wow. Drop the mic.

I think the whole reason digital CXO exists as a site is for what everything that Steven just described. So by all means, come check that out folks. It's one of our, uh, pet projects that we're kind of fond of, but we think that ultimately the audience for it is gonna be everybody.

Absolutely. Alright, let's take a break. Second break here on Text Trunk Gang.

We're gonna come back with a study outta Stanford around the RAG. Stevens recently had some exposure to this. I hope he's not contagious, but he's gonna tell us all about it.

You're watching Textron Gang. All right, we're back. And we're talking about this Stanford study that looked at the implications of RAG for, um, improving or at least making the hallucinations that we see in Gen AI platforms a little less problematic in that we, I saw the hallucinations.

I thought this was A Colorado story or something. Oh, yeah, yeah, yeah. There you go.

Well, you know, maybe, we'll, we'll get Mitch's opinion in a bit. I'm seeing things clearer now that I'm not in Colorado. There, there, there you go.

But the, but the premise of this study is that, and it's been an ongoing conversation, is that we do have these hallucination issues with LLMs, and the theory is that if we expose them to more data, we'll get less hallucinations. And the study seems to confirm that, but it doesn't say that we can eliminate them. So let's start with Steven here, but, um, what's your take of what's going on?

'cause I know you've gone to a couple of conferences where a lot of folks are talking about this kind of issue and what's happening, but is this something we just need to learn to live with? Or are we, is it gonna get better? What's your assessment?

Yeah, I'm thrilled that I was able to go to, uh, this Click Connect conference and I was also able to spend a lot of time with Google and with Amazon talking about exactly this topic. And, um, in fact, on my podcast utilizing tech, we are talking about AI as our focus. This, this season and, and RAG has been, uh, certainly an improvement.

And it has been touted as basically the, the, the, the catchall fix all for the hallucination problem. Ultimately it is not and it will not be. And that's what this study shows.

This is a very, very sharp study and I was so pleased to see it in mass, uh, you know, fortune in, in a mass market publication. Because on the one hand it talks about the value of, of RAG and how RAG can reduce hallucinations, especially in the legal field. But on the other hand, it also points out that that's not the, uh, fix all, uh, for this problem.

One of the issues so far has been that people are literally talking to an LLM as if it has intelligence. I don't know how you're gonna define intelligence, but it doesn't LLMs no matter how complicated they are, no matter how good they are and how many parameters they have, they don't have the same kind of intelligence that humans have. They will not have the same kind of intelligence and they will not be able to produce, um, halluc free hallucination free error free output at any point.

Full stop. An LLM is a user interface, mark my words. An LLM is simply a human computer interaction element.

And if you approach it from that perspective, it can be incredibly valuable. We saw that, um, for example, with Qlik, they have a new product Qlik answers that is basically a rag plus LLM, uh, tool. And you can feed it a subset of corporate documents, either structured data or unstructured data, ask it questions in a copilot type scenario, exactly like they were talking about in this article.

And it will produce results that are constrained by the data that you feed it. So we actually tested it last week in person and we were asking it nonsensical questions and it refused to answer those questions. We asked it smart questions based on the dataset that it had, and it answered those questions in a convincing manner.

And you could say, well, full stop, they've solved the problem. But the trouble is that they haven't, because ultimately RAG gets to most of the way there because it's not gonna hallucinate something completely nuts, but it is still not a person capable of reasoning. And so in the example here for this article, um, if you connected an LLM to Lexis Nexus and asked it questions about legal issues and said, you can only get information from this, from this document set, you would get much better answers than just asking it without any kind of data sources, uh, for legal citations because it's just gonna make stuff up.

In that case with Rag, it's not gonna make stuff up, but you can't then go to the next point, which is to say that it's gonna be a hundred percent correct because it absolutely is not, it's not gonna give you the best citation. It's not gonna give you a complete citation, and it's not gonna give you a full understanding of the legal issues because that's what people are for. So I wanna jump on this and give an alternate point of view.

Agree with you from a rag perspective. The value that it can bring, it can help to reduce halluc hallucinations. But there's a mindset that we have to take as well, which I think is well, might pique your interest.

Think of hallucinations as a feature and not a bug. So if we're trying to get to absolute perfection when we're generating code on behalf of the human, okay, there's a problem. We're generating the legal case on behalf of the human, we are gonna see problems because it's not, it's math under the covers.

That large language model is math. So if you leverage it though in different ways, think of those hallucinations as creative muse, it actually can be really, really insightful. Ideation and brainstorming.

We're leveraging it with teams right now where we have people working on requirements and they're taking in user stores, they're taking transactions or, um, transcripts from conversations that they're having. And it actually can be like a muse. Same thing when we're talking about test plans and having people really think outside the box.

It can be fantastic for ideation, um, experimentation, some great things there as a conversation agent, but all of it is, as you're talking to do that, that creative person that, you know, that friend that you talk to who has all these whack-a-mole ideas, and some of them are good and some of them are not so good. So there, there are different places and reasons and ways legal, um, what we're talking about with LexisNexis. Yeah, we gotta get things right and rag will only get us part of the way there.

It is actually constraining the creativity, the creativity, the mathematical creativity of the language model in order to help us to get more reliable outcomes. So just, you know, just an alternate point of view to really think about. Guys.

Yeah, To Tracy's point, I'd love to get your opinion on this. 'cause I think we've talked about it a little bit in the past, but humans tell me things convincingly that are wrong all the time. And I don't call them up and accuse them of hallucinating.

I just call 'em up and say, oh, I do, I, I word call 'em up and go, Hey, bonehead, that wasn't true. And when we move on, No, I just said, are you on drugs? Are you hallucinating?

Yeah, no, I've said it before, but yeah, I understand the point you're making, Right? So is the machine just one more Source? So should we be kinder to our machines, say, Mr.

ai, are you not feeling well today? Today, Yes, exactly. Because you know what?

All These things, it's tied to your AI week. We don't, someone will make a holiday over this. We don't trust all the humans we deal with.

So why should we trust all the machines we deal with? What makes you think the machine's gonna be any better one way or the other? Because well, if what if the machine's knowledge is based upon what the human knowledge is and the human knowledge is flawed?

Well, of course the marine, the machine knowledge is flawed, if that makes sense. But I think the whole point around AI is we want them to be smarter than us, better than us, not make mistakes. We've gotta be, if it's not reliable, it's not useful.

We, we might as well just let humans do it. Isnt that unre unrealistic expectation though, isn't What you saying? I'm not saying it's not I, well, let me, it's an unrealistic expectation today, but I think the expectation is going forward five years, 10 years, whenever, and we get general intelligence and all these other things we heard them bandy about is that you won't have the hallucinations or are we destined to always have Hal 9,000?

Well, The problem is we're never gonna have that. I'm sorry. We are never going to achieve artificial general intelligence that matches human intelligence in a similar Way.

Don't exceed. I think we can get super intelligent ai, but it will never be like human intelligence, in my opinion. Um, and furthermore, you know, to to, to Tracy's point, I I, that's, I I love that insight.

It is a muse. The problem is people keep re creating, they keep treating LLMs not as the middle of the process, but as the end of the process. They keep treating LLMs as the, the, the, the answer machine.

Mm-Hmm. Not the question machine. And, and, and it's right there in the names.

I mean, you got Google putting it on the, on the answers, you know, Google answers. It's the answer. It's the, it's not the answer.

It is, it is an idea. It is a muse, as you said. I love that word, that term.

It is part of the process. It is valuable, but it is not the end. And it's going to vary by each one of the, I'm sorry, Mitch, to step on you.

Just to add to that, and then I'll, I'll pipe down. It, it all comes back to context. As we've been talking about many, many different topics.

How am I using it? Where am what's my intentionality for this particular use case? For this particular part of software or the domain that you're working in, uh, in marketing, sometimes it can be closer to that end stage.

It can be really, really far towards the end of that workflow, right Steven. But it is not always at the tail end. So we've gotta figure it out.

What's my risk if it's wrong? That's what we need to be asking is in this case, what's my risk if it, if this is wrong, and what do I need to do to protect against that? Do I need more humans in the loop?

Do I need to add rag? There are other techniques that are coming out and growing, right? It is it one of those other techniques?

It depends on the use case. It really, really does. Mitch, I think we have a paradox that we're faced with, which is we chase the rabbit of we need more data that will make our LLMs accurate.

Well, those large l LLMs are fed, you know, massive amounts of data. We know that that data has a lot of errors and inaccuracies just complete absolutely wrong information in it. And at the same, so we're chasing the rabbit of, of, uh, how many NPUs and cycles do processing?

Do we need to train those models? How much data can we just feed it more data? But at some point you have to look at the data going in, you know, the old garbage in, garbage out kind of thing.

And we call that a solution or hallucination instead of garbage out. So if you want more accurate, uh, more accurate responses out of ai, you either have to invest in algorithms can discern between accurate inaccurate information. Maybe it's certain number of sources that it can validate to determine whether this is a more proba, probabilistically, math wise, a a more accurate answer than another.

But I, I think we still have to look at the quality of the data going in. 'cause we're just sucking things off the internet. Um, I think the in internet's a hundred percent, right?

Last night time I checked it. So I, I think it's all about the data. It doesn't matter how much processing or algorithms we applied to it, if we're still giving it crap data, we're gonna get crap answers back.

That's my curmudgeon, my bizarre curmudgeon answer. I agree. Amanda, I wanna, lemme get Amanda in here for a second.

Guys. It's not all about the data, it's about the data and the type of intelligence and the expectations we're placing on these systems. Yep.

I would agree. Amanda, Table Found in agreement. Let me get Amanda in here for a second and ask her this question, right?

What if the output is something that doesn't align with my core belief? So I may believe that the world is flat and therefore, you know, the machine is hallucinating when it tells me the opposite. And so how, how will we align our personal truths with Whatever, pretty much how our country runs today, doesn't it?

Yeah. Well I think that's part of the problem we're we're talking about is the AI is only gonna give us what we give it. It's only as good as we are.

So our biases are, are part of the ai and we're going to read into it what we want, just like we do with everything else. Well, can I bring up something that maybe this will be a, a future conversation that we have? I, I was just reading about Open AI is looking for more data sources.

So they've, uh, they've approached hand selected areas, hand selected media, hand selected news organizations, and they're entering into contracts with them. But they've also said that they won't talk, they won't be entering into conversations with what they consider to be fringe or not common points of view left, right. Doesn't matter.

What happens then to that LLM if we're, if we're censoring the information that goes in censoring the different points of view, are we going to lose creativity? Are we gonna homogenize over time? Because we said we're just gonna get that from the Wall Street Journal and the New York Times and Fortune magazine.

Those are our three big additional data sources to help round this out though Actually you're saying that if I, if I restrict information to my generally intelligent LLM, I'm violating its rights. That's what you're telling me. I think that that's an interesting way to put it.

Amanda, you've got something to say on this More, it's a filter. Yes. I think, I think it comes down to that's when we talk about the more data there is, I think the more and more data from all sides and points of view, the more accurate it's gonna become as it can filter through all points of view and All data learns to filter through.

People can't filter through all the data that we get here on social media and everything else. I don't know how we can expect our ais to, but guys, whether this is a hallucination or not, we, we've gotta pull the plug on this, this episode of Text on gang. I'm just Mitchell, enjoy.

I'm, I'm just checking to make sure you're actually a person. Well, it goes right through, right Mitchell, enjoy AWS reinforce. Steven, great to have you on.

We hope to have you on here soon. What, what's your next tech field day, by the way? I think it was cloud.

You said Cloud Field Day. It's actually Wednesday. Um, and in fact, tomorrow, if you're interested in this topic on Thursday, we've got Google Cloud, uh, talking AI all day Thursday on, uh, Textron tv.

I believe It'll be on. Absolutely. Thank you Tracy and Amanda, we'll speak to you soon.

Keep up the great work, Mike. Great job today. We will be back, I guess on, uh, Thursday with a fresh tech strong gang reminder starting, I think it's June 18th or 17th.

We go five days a week on the gang. All exciting stuff. Until then, though, there's Alan Shimmel for Techstrong.

Have a great day everyone. com is the leading resource for news analysis and education on challenges facing the cybersecurity industry. com covers all aspects of cybersecurity, including data security, DevSecOps, cloud security, application security, network security, security threats and more.

com has the largest selection of security content featuring breaking news, blog posts, podcasts, and more. com to learn more. com.

Home of security bloggers network.