The Gap Between AI Promises & AI Reality – Utilizing AI Episode 24

Even as large language models have become incredibly powerful, there's a backlash from end users looking for real-world applications. This episode of "Utilizing AI" features Scott Robohn from Solutional and Olivier Blanchard of the Futurum Group, considering the progress that has been made with AI agents and the progress in delivering actual value. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence from the Futurum Group.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Stephen Foskett, President of the Tech Field Day business unit here at the Futurum Group. Before we dive into this discussion, let's meet who's on the panel today.

Hey, Stephen. I'm Scott Robohn. I'm the CEO and co-founder of Solutional, and we're a consulting organization that helps our customers rationalize the adoption of AI in their processes.

And of course, I'm Olivier Blanchard, research director at the Futurum Group, and my focus is on AI devices. So basically, anything that's AI-powered, AI-enabling, that's not bolted down to a data center, is part of my coverage. So it's good to be back.

Great to have you both here. Scott, you and I have been friends for a long time, and we've had a lot of conversations about AI, and to misquote the real world, find out what happens when people stop being polite and start getting real. Well, that's what's going on here, right?

AI has been exciting. It's been something that a lot of people are enthusiastic about. But I think that there's an increasing acknowledgement, both on the corporate side as well as certainly in the general consciousness, that we need this thing to get real if it's going to be real.

We need there to be productive applications. One of the most scathing critiques I've heard is that of all the development that's happened on AI, the only really compelling development that people have seen are basically AI supporting AI, not AI supporting the rest of the world. And so I guess let's start with that, Scott.

How do you answer that critique, and what are you seeing out there? Yeah. I think with any introduction of new technology, we always have to endure the hype cycle.

And so on one hand, that's not new. On the other hand, there's been at least a 10X of the hype with AI that we've had to endure and that we're all trying to sort through and see, okay, where am I going to get value out of this? Whether I'm a builder and I want to roll my own, I'm going to create my own agents that fit in my environment, I'm going to stand up my own MCP servers, or I'm going to look at vendor options, shrink-wrapped platforms that have ready-to-use agents for me.

We've seen some early examples of the shrink wrap piece where some of those agents have been pretty clunky. I won't name names here, but you can think of your favorite meeting assistant type tools where maybe the workflow for getting an agent involved in one of my recordings is very unintuitive. There's room for improvement there.

I do feel like we're starting to see improvements with what the vendors are delivering from an agent perspective and people who are doing their own and understanding more capabilities. Let me give you one specific example. There's this long-standing software development concept that I'll misquote horribly or oversimplify.

Do one thing well. Right? So instead of having agents do a bunch of different things, let's create agents that do a subset of things very well, that are very focused, and that can be orchestrated.

And I feel like that's starting to catch on in the network operations world where I live, but also in other domains. Well, no, that's a really good place to start. I feel like there's definitely some disappointments.

We're at the disappointment stage of the hype cycle, where it scaled very quickly. We were suddenly able to, overnight, have access to all of these different AI products that allowed us to do really cool things like enhance search, sort of like intelligent search results that are catered to you, and then creating videos and images and doing all these fun things, generating documents, doing live transcription, all of those things. And then it just sort of, every couple of months, it felt like it was just getting not just marginally better, just significantly better.

And now here we are, and it feels like AI has sort of plateaued a little bit. Hmm. " Right?

It's my agents or the agents that I was told were coming aren't very good, or they're just not really working at all. There's not a whole lot of customization that I can do, or at least not easily. Most people aren't techy enough to really be able to set up their own servers and do these things.

They probably don't have the budget either. A lot of the better features are sort of behind a paywall. They're premium.

You have to pay extra for that. And I feel that part of that touches on what you just said, which is the AI products that I see that are still being used with enthusiasm are the ones that do one thing well, whether it's video creation, rendering something, the transcription piece, helping you summarize or think of an action plan for reports, doing fact-checking, things of that nature. And it feels like, and I've always said this from the very beginning, that the real next frontier was orchestration.

It wasn't the AI that can do it all. It's just ultra-specialized AIs and agents that can be orchestrated together as a network or as a team to be able to perform different functions simultaneously. And that's the piece that I still find vastly missing, especially in an ecosystem where every AI company is trying to build the AI that can do it all.

Sort of like the Swiss Army knife of AI products or AI capabilities, as opposed to just focusing on things that do one thing really well. I feel that one of the advantages that Anthropic has had recently is that it had initially that focus of trying to do a few things extremely well, where other companies, like OpenAI, for instance, was trying to focus on doing sort of being a general AI that is adaptable to a lot of different use cases, but without doing anything particularly well over time. Yeah.

So I'm curious, to throw it back at you, Scott, how you feel we can move forward with this and where the next sort of ingredient in the secret sauce really lies. Yeah, that's all fair, and I'll give you at least two vectors of response. On your comment about plateau, I agree.

And we need to catch our breath a little bit, right? Seeing the plateau, or at least perceived plateau this week , let's leverage it, right? And it gives people a little bit of a chance to catch up and say, "Okay, I don't have to worry about the next change for the next model, or should I flip from Codex to Claude Code," or pick your favorite coding tool of the day.

So I think the plateau is good. I think we're also seeing other things emerge, like spec-driven development, right? That's kind of emerged on the heels of vibe coding, where vibe coding was taking up all the oxygen in the room, and is really good for prototyping and ideating.

But spec-driven development, bringing more standard product development processes to leveraging agentic coding. Like define requirements, turn it into a spec, and then let your agent or agents loose on a spec with other constructs, too. I'm super oversimplifying it, but I feel like the plateau is helping other things mature in this space.

Yeah, that's an interesting point because first off, thank you for bringing up this spec-driven concept because, of course, I hadn't really thought of it in those terms before. But if you think about how people configure, let's say, OpenClau, it is actually almost spec driven in that essentially you're creating a markdown file that describes what you want that agent to do and how you want it to do it. So it is an entirely opposite of the context that Olivier was bringing up, which is this sort of, this thing is amazing and magical and does it all.

And I agree with you. I was just at Qlik Connect, a conference for data and analytics pros. The conversation there around agents was very much what the two of you are saying.

Essentially, these people are extremely jaded about general purpose tools and extremely focused on the idea that this statistical analysis technique that is large language models could be very useful for this and that and that thing over there, and so let's try to make a thing that does those things. And I think that's a lot more healthy. I think it's a lot more realistic, and it's a lot more likely to have real-world benefits.

I understand if I was the CEO of OpenAI, I would want to make it sound like I was building the HAL 9000 plus the bridge computer from "The Enterprise," and maybe not the Terminator, but maybe, I don't know. Because I would want to make it sound like my technology could literally take over the world and result in massive job displacement, at least. Maybe not job cuts, but let's say just job displacement.

Sure. Again, I'm pretending to be the CEO of this thing. I don't know.

Maybe I'm not stupid enough to say I'm going to fire half the world. But they seem to be. But the point is, I might go out there and say, like, "This thing is omniscient and artificial general intelligence, artificial super intelligence.

" That's just good marketing. In reality, this plateau that we've described, which is absolutely true, what we've seen is definitely a case of diminishing returns as these models are trained and trained and trained, that they're getting sort of asymptotically closer but not really touching actual intelligence. And that's okay.

Maybe we have to say, "Wait a second. Maybe this is not HAL 9000. " And trying to take advantage of that capability.

I think that's kind of where you were going, right, Olivier? Yeah, it was. And I write for a living, I do a lot of analysis for a living, and of course, I have to use AI just to see, for my own personal productivity, but also out of curiosity, so I could write about it halfway intelligently.

And I have to tell you, I'm reluctant to call it AI at all. It's not really artificial intelligence. Hear, hear.

I'm annoyed. The rest of the day we're doing a lot more heavy lifting. Yeah.

Yes. It really does. We're not quite there yet.

And what I've found is that in terms of just as a user, right? So this is purely anecdotal. I'm not sharing any data or any research on here.

I could, but I'm not going to right now. What I'm finding is that on the one hand, there are parts of my job that are significantly easier thanks to these tools. I can synthesize an article or a story better.

I can bring sources together. I can sort of create a really good first draft much more quickly thanks to the AI tools that I use. And so that saves me time, right?

But I want it to be really clear to managers and business leaders who think that AI is going to solve productivity problems, that all it does is really shift those time resources in other directions. Because now what you have to do is you have this first draft, you have to edit it, you have to fact-check it. You have to do all these things that you didn't have to do before because your process was different.

And so in the hours, say it takes an hour to write a story or an article or a research note, where it used to take you an hour to do your research and write your first draft and edit and do the things yourself, it still takes an hour. You're just focusing on different things, on different tasks in different ways. And so you're shifting the load, but you haven't actually become more productive.

And for me, the value of an agent is that productivity, whether you're doing more in your one-hour time block, or you can do the thing that used to take an hour in 10 minutes, and now you have 50 minutes of free time, either to do other things or to do nothing at all, to improve your quality of life, to go for a walk, whatever the equation is for you, the value equation. But I'm not quite seeing that yet. And what I'm disappointed by when I look at the way that all of these AI companies are sort of positioning themselves and marketing themselves, is that it's all about scale, and it's all about speed, and it's all about building things and building infrastructure and future-proofing, but it's not about actually delivering ROI and results now.

It's not really about doing anything particularly well. " It creates an environment where you're spending a lot of money, you're driving a lot of resources towards a future outcome, and you're creating an environment in which, at any point, a large segment of your customer base could jump ship because you haven't really built that pipeline. You just created a product in the moment that could become obsolete next week, and then suddenly all of that spend, all of that effort, all of that time is just wasted, unless you can sort of reacquire those customers later.

I find that incredibly dangerous, and not in line with the way that we've talked about or thought about product development ever. And to come back to Scott's point about spec-driven development, as a former product manager myself, who has participated in the development from ideation to marketing of physical products, it's really important to go through these processes and not just sort of build out this magical infrastructure that somehow will generate unlimited revenue forever just by virtue of existing. I feel like we really need to go back to, okay, what problems are we trying to solve, and let's solve them, as opposed to let's build a technology that will magically solve all of our problems in 10 years.

Yeah. When vendors say magical or imply magical, shields up, captain. Or to misappropriate the Marvel Cinematic Universe, with great power comes great responsibility.

And there are a couple things that each of you have said that strike me on that front. So looking at the utility of how you're seeing the tools help you, Olivier, I get that. I see it personally, too.

I think we can all also think about personal examples where we let too much slop slip through our fingers or into final product. I'll admit to that. And that's another skill that we have to learn, to not be overly impressed with the nice formatting and the stuff that sounds confidently wrong.

We still have to be the final check on what goes in our blog post or what goes into our client reports. We're not there yet. You still need to be ever vigilant to check those things.

I'm not saying that's good or bad, but I'm saying that is what it is, and we need to address it very clearly. The other thing I would jump on is when you talk about displacement, Steven, againWith great power comes great responsibility. We have to admit that openly.

There will be real displacement. We can't put our head in the sand on this. I go back to the switchboard operator.

Was the switchboard operator displaced by mechanical central office switches and then electronic switches? Sure, absolutely. Did the demand for telephony services decrease?

Absolutely not. So both of those things can be true, and we just have to figure out how to balance them. Yeah.

And on that note too, if we can kind of get a little bit specific here, I actually had a conversation over the weekend with a whole bunch of open source people, including Bradley Kuhn, who is a software freedom lawyer from the Copyleft and Software Freedom Conservancy, about some of the topics that we've talked about here. Now, both of you now have hit on something I think that is very important with regard to AI. So what we were talking about this weekend was the copyrightability, or lack thereof, of AI output.

And he corrected me, schooled me, in some of the current understanding of copyrightability and the impact of Supreme Court cases and so on, on AI. Among those, though, was basically the contention that both of you have gotten to, which is essentially that AI plus humans equals content, equals art, equals output, equals copyright. We need to make sure that these tools are tools.

And as long as they're tools, then what they produce is an act of human expression and is thus copyrightable. Specifically what we were talking about was the idea that if AI is contributing to open source products, for example, can that code be licensed at all, let alone can that code be open source licensed? And the answer is probably yes.

But as long as people are involved, definitely yes, because essentially, it becomes that person's expression. It's the same with Olivier writing a report using AI as a way to collect information and organize information, and maybe even help work through the thought process. It's the same with Scott, with whatever software development that you're doing.

If you're writing specs and using this tool to help, then it becomes-- People have sometimes denigrated it as a glorified autocorrect. But in a way, it kind of is. Software coding assistants are properly used as glorified autocorrect.

Essentially, here's my thoughts, here's what I want to develop, here's some code that I can then work through to help build something. I think that's realistic. I like that idea.

Olivier, is that kind of the direction you were heading? Yeah, pretty much. But also, I think a lot about Alexa- Mm-hmm ...

and Alexa devices. Don't say that too loud. She's going to talk back.

Yeah. Sorry. I've had to change mine from- I have my headphones on, so yeah ...

yeah. The wake word I've changed to something else, which you can. There are a few options.

I wish there were more options. But I think that there's been this promise that's sort of faded in the last few months. But a year ago, we were talking about, or hearing rather, from a lot of vendors, a lot about hyper-personalization, which is a really great marketing term.

But the promise was essentially, look, think about the A lady or the S lady for the Apple ecosystem, or Gemini, or probably just triggered all of my devices. But there was this sense that you would be able to build your own agents, give them their own names, their own skin, and create sort of a, again, to borrow from the Marvel Cinematic Universe, a Jarvis-like ecosystem. Basically, the same thing that Tony Stark has.

But even potentially give all of your agents different names. So you could have one master agent that is sort of the big orchestrator and sort of leader of all the other agents that are under that agent, and ask that agent to do XYZ, and then it just finds the agents that are specialists in that particular task. And they do it, and that main agent is the interface.

Or you could go directly to Rosalind or Heather and have them, Jack, and have them do grab your sports scores, make a purchase, make a payment, look for hotel reservations. Do all of these things verbally, almost frictionlessly, wherever you are, if there's a device that's sort of passively listening to you and listening for those prompts, that they'd be able to do those things for you, and that you could train them, and they could be all secure, and you could even have an AI hub in your house that provides an extra layer of security. We're not seeing that.

I'm still working with the same tools and the same little dialogue boxes. We're starting to see more voice-first interfaces, but it's really not moving in the direction that I thought we were going to be moving into by this time this year. And I have concerns about that, because I don't think that is where the AI product teams are focusing, because I don't know that they know how to monetize that.

And it's a bit of a chicken and an egg problem where do we need the hardware first? But nobody's really going to focus on the hardware first if we don't have the software layer that's sort of moving in that direction. And it's going to take a company like Apple or a company like Google to build this out because they have that full stack.

But for everybody else who's in the ecosystem, you're either a hardware player, a software player, a platform player, and if you don't have the full stack, you're not going to invest that much money into it. Yeah. And I think that's part of the problem.

Yeah. Yeah. A brief follow-on to that would be, there is a world in which AI will become more of a how than a what.

You have product management experience, you have a set of tools that your engineering team is using, and you don't really care what tools they use to implement certain functions and features. Right? We're going to start to see AI become more of the how and not so much of the what.

I'm not saying the what part's going to go away, but the how is going to become much more important. Yeah. Yeah, I love that because that's what we should have been thinking of this whole time.

Yeah. " Right. Yeah.

I remember, I'm going to give John Willis credit for that. I don't know if that's- Sure ... really him, but it was somebody like that.

And he was definitely standing there when I heard that. And I think it's the same with AI, right? Right now, we are absolutely flooded with the hype for this technology.

Soon this is going to be another tool. It's going to be an incredibly valuable tool, and it's going to be one that's going to find itself a place. And just like a lot of other revolutions where we've desperately tried to figure out where it fits, we're in the midst of that.

But I do wonder about this chicken and egg thing because one of the challenges that makes AI different is that this thing is incredibly resource intensive. It's incredibly demanding. Olivier, you are our mobile device monster here, and you know just how challenging it is to deliver AI features on the go in the way that people want to consume them.

Without the risk of opening up another can of worms here, is this technology really going to ever be practical- Yeah ... on the go? Yeah.

It is. We're getting there. And actually thinking about spec-driven development and design, we're starting to see that.

So when I talk about these full stack players, right, that create their own silicon and also have the software and have the entire stack, the whole channel, they're able to do this. And so we're starting to see this with Google, the Pixel ecosystem, and the Fitbit's product line, which is part of Google, has been absorbed by Google. I think you're going to start seeing some things in the next few months, can't talk about it, that are going to drive towards that.

I feel like the silicon that's being built is custom-built to be able to handle these orchestrated, sort of partially local or on-device, and partially in the cloud functionality with this orchestration of okay, what is the best way and where's the best place to do this compute? But also, I think it's important to note that training and inference are very different and have very different requirements in terms of resources. And so once you've trained a model to a point, then you can make it much, much more efficient.

And the inference piece of that can be handled very locally. And so you are going to start seeing a push, or an acceleration rather, in mobile devices, in PCs, in wearables, where the local inference capabilities are going to be significantly improved. And you're going to start to see these devices become much more functionally potent in terms of AI.

Yeah. So it's coming. But that's what it takes, though.

It takes the Qualcomms and the Intels and the Broadcoms and the MediaTek's and the Googles of the world, the Apples of the world, to create their own silicon specifically to align with the software, in order to be able to deliver a complete product that is very spec driven, as opposed to hardware and software and AI infrastructure sort of trying to figure out where the affinities are. So yeah. And I think we're going to see much more success in those very pointed, specific use cases, than we're seeing anywhere else.

Yeah. On that point, I kind of agree with you, Olivier. What we've seen on the hardware side, I believe, is generations now, rapid generations, trying to figure out exactly what hardware we need in order to handle this.

Personally, I've got a stack of AI hardware, and I've got a Jetson TK1 sitting over there that's completely useless, because they weren't sure what kind of hardware we would need, how much we would need, et cetera. And we've seen on the Qualcomm, Apple, Intel, AMD, MediaTek, all these companies, each generation gets closer and closer to what the software actually needs, to the point that I think that the latest generations are actually much more capable of doing this work in an efficient way. And that's all happened in just a few years, just a half a decade here.

So Scott, what do you think? What's your summary here of our AI getting real moment? So there's so much more to talk about the hardware and the platform stuff.

I can't wait for the follow-up on that. But to encapsulate what we've talked about today, we see this pause, we're catching our breath, we're figuring out what's useful, and I think I would summarize all this by saying, we are going to collectively find where the utility is, and even if we find utility in it, at scale. " We're in this period of discovery.

I think we're going to be here for a while, and we're going to be thinking about what do these energy constraints put on us. I see it as an exciting time, where we are working out some of the very important details, Steven. Absolutely.

And if I can be self-serving, that's kind of what we're discussing here. That's why we started this, that's why we got weekly updates here on the real world of AI. And if you've listened to this and you said, "Oh my gosh, there's more that they could be talking about," you're absolutely right.

Olivier and Scott and I have been saying the same thing behind the scenes here. I think almost anything we brought up here could be another hour of discussion, and hopefully it will be. We post these episodes every Wednesday.

Please do tune in every week for this, so that we can continue this conversation. Scott, where can people connect with you and continue speaking with you about these things? Feel free to hit me on LinkedIn.

I'm pretty good at responding to any kind of inquiry I get. May not be a 24-hour turnaround, but you can also see me at a couple different events in May where I'll be talking about AI in network engineering, and I'm really ramping up for the next Network Automation Forum event, AutoCon5 in Munich, Germany in June. So if you'll be there, talk to me there.

Cool. And for me, X and LinkedIn obviously, but I'll also be at Dell Tech World, that's coming up next month. And also Amazon Ads in New York in a few weeks.

So if you're in the neighborhood, hit me up. I'll talk to you. Yep.

And for me, by the way, we do have our AI Field Day event coming up here in a couple weeks, May 13th through 15th. ai, or basically any kind of social media. You'll see live presentations there from companies talking about how they're really using this technology in production.

Thank you very much for listening to this episode of the "Utilizing AI" podcast. If you enjoyed this discussion, please do subscribe on YouTube or in your favorite podcast application, and give us a rating and review if you can. This podcast was brought to you by the analysts and experts from the Futurum Group, where insights meet AI.

ai, the "Utilizing AI" YouTube channel, or the Techstrong TV app. Thanks for listening, and we'll see you next week.