Balancing the Relationship Between Humans and AI Agents – Techstrong AI Podcast EP44

Hello and welcome to the Techstrong AI Podcast. I'm Amanda Ani, and with me today I have Andy Lynn. He is the CEO of Provoke Solutions and an AI expert.

How are you doing today? I'm Doing well, thank you. Can you share a little bit about your business and what services does provoke solutions provide?

Absolutely. So Provoke Solutions is a international company that provides technology and management consulting services. Um, so largely the work that we've done historically over our 23-year-old history, um, has been focused on custom application development.

Um, and you know, obviously since the arrival of cloud and really focused on cloud, uh, based development. Um, and in the more recent times with, uh, the arrival of generative ai, we've really shifted, um, how we deliver our services, uh, to leveraging generative ai, uh, via, you know, AI agents. We have a piece of ip, uh, that embodies that.

And that IP is actually currently being sold to market as an independent product that customers can use on their own with, you know, some light professional services support to clients who still engage us as, you know, a traditional services engagement would be, but instead of all the work being done by, uh, you know, humans, um, as resources, we are shifting a lot of that work to be done by agents, uh, what we call digital coworkers. Awesome. Alright, so we're gonna talk a lot about AI agents.

We're hearing so much more about Agent Toki these days, and I have that my topic is why Gen AI tools often miss vital elements of the project scope requiring developers to provide business specific adjustments. Can you explain that? What do you mean by that?

Yeah, so what, what often happens is, um, everybody kind of starts off a project with, uh, the perception that everything is set. It's very clear what needs to be built, what the success criteria is, and then fairly quickly, uh, as you get into even requirements refinement, um, or you start doing the development work, you realize, uh, there's a bunch of questions that haven't been asked. Uh, there's a bunch of answers that aren't very clear.

Um, and so you go through this process of trying to get that clarity. And then there's oftentimes that we, we run into this quite a bit as a company where the clients don't actually know, uh, what it is they want to build, right? Or if we're modernizing an old system, they don't know what the rules of the old system were because the requirements have become outdated, they've been lost.

And so sometimes one of the first things that we need to do is to help the customer understand the, what, what is it that they want to build? Um, and in the traditional way, right? Like you, if you had an existing system, you'd, you know, get a, a technical person and an analyst and they'd sit together and they pour through all the screens and trust to reverse engineer the requirements.

And that is, you know, painstaking, soul sucking type of work. You know, that's not the work that everybody loves to enjoy doing. Uh, but with gen ai, specifically our product called Nova, we're able to go in there and look at the existing system, look at the ui, interact with it in an automated fashion, and be able to produce a cut at what we think the requirements are, right?

And now it's just a matter of, you know, a business subject matter experts reviewing that and saying, this is right, this isn't quite right. And oftentimes it's really revealing, like one of the most satisfying things is when you see a customer go, oh, I didn't know the threshold was $20,000. It should have, we always thought it was at 50.

Well, your system has it at 20, so we're gonna be modernizing this. What do you want it to be? Do you want it to be 20 or do you want it to be 50?

Um, and so it's that type of clarity that I think, um, is always missing The beginning of a project and an example, have a, a agent AI can really help, um, lessen the impact of that type of a gap, um, in filling that gap. Awesome. So once it comes, once they've come to this realization and they're looking into new technology and, um, projects to start, what advice do you have as far as implementing the new technology and making sure everybody is on the same page with this change management?

Yeah, so I mean, I think it's, um, you know, the, the arrival of gene AI I think is more like the arrival of the internet, right? And so I, I think I see a lot of our clients struggling with questions that there's currently no definitive answer. Um, and so you have a choice.

You have a choice to say, well, I'm gonna move forward and experiment and build our own lessons learned and our own capabilities, or you're gonna wait until things are answered. Um, and I think you can see the winners from the internet age back in the, you know, late nineties are the ones that said, Hey, you know, we'll figure this stuff out later, but let's do what we can start experimenting and build from there. And I would say that's probably the same type of approach.

I would, uh, try to get executives at organizations to espouse, right? There's a thousand questions around data and security and privacy, but there are ways that you can work around that. So involve your chief information security officer early, right?

They will have a perspective, but the answer rarely is let's stop until everything's figured out, right? Like, there's legislation that needs to happen, there's corporate governance that needs to happen. There's bad practice that need to develop, but we just have not as an industry had enough at bats to really be definitive of what those answers are.

Um, so like everything else, it's, you know, being measured about experimentation and really making sure that you're learning and growing within guardrails, uh, making sure that cross-functionally you've spoken to and gotten everybody across New York bought in. This isn't just a business thing. This isn't just a technical thing.

This isn't just a security thing. This is a company thing and representatives across all the various functions of the companies need to come together. Legal needs to be involved, right?

Um, 'cause a lot of the concerns that I hear are actually, well, they're not really concerns, they're concerns if you use the public facing, uh, LLMs, but like everybody else that has cloud, if you have your own private instance of an LLM, your data leaking, your security concerns greatly, greatly reduced. Um, so it's a lot of conversations and, but having that innovation spirit that we're, we're still gonna go out and try some things, you know, find the right risk and reward balance and we're not gonna let it paralyze us, is I think the advice I would give. Okay.

So we're hearing so much about agentic AI and using AI agents in the workplace. First of all, what advice do you have as far as skills? And are are we seeing that employees are lacking some skills and what training should they get?

Y Yeah. Um, again, the shift, uh, you know, from on-prem to cloud presented similar challenges, right? Do we still need system admins?

Do we need network engineers? 'cause everything's moving to the cloud. Um, and the truth is there's more it and tech jobs now with cloud than with on-prem.

So the industry has grown. So to your point earlier, it's not that I think jobs are gonna be reduced, I think it's just they have to make a pivot. They have to make a shift.

Um, they have to learn new skills. Um, I mean, I'll just be very authentic and personal. My, I have a son who's in college now, and he's like his dad.

Like I, I grew up as a software engineer. I've been developing since I was 10 years old. And, you know, he kind of picked up those genes.

He's pursuing a computer science degree now, but what he's seeing what's happening in the industry, right? And so he's now pivoting to what might be something that can align to his passion, but still be useful for the industry. Um, so, you know, I would say the, the skills that people need to learn are more prompt engineering, right?

It's more how do you interact with the system? What are the right questions to ask, right? And in a lot of ways, it's forcing us to be much more analytical where it's not having the order given to us, like we have to construct the right questions to define the order.

Um, so I think, you know, whether you're an engineer, a business analyst, a creative, uh, a strategist, understanding prompt engineering, um, doing that effectively, having the resiliency to say, Hey, that didn't work. How do I refine my prompts? Uh, is probably going to be the most critical skill that everybody develops.

Um, now does that mean all of the skills that we've developed over the last 20 years are absolutely innocent now? Um, you know, there's always gonna be a human in the loop. Uh, I'm not sure we're anywhere close to the time where you don't need any human supervision and, you know, the agents are just gonna crank out outputs and without any review, we're just gonna keep letting them go through.

I don't believe that, uh, I don't think that that's possible. And then as we start, you know, having agents, as we get comfortable with agents solving problems that have been done over and over and over again, right? Deterministic stuff, eventually you're gonna run into stuff where, look, agents rely on a large amounts of data of work that's been done in the past.

But if we're trying to break into new territory stuff that's never been done before, agents won't be able to do that, right? Like, that's not gonna be their strong suit. And that's where humans are still gonna continue to play a role.

Um, so I, I think the two things that I'm training and getting my team, um, evolving to is, yes, you still have to keep up with the tech stack and the latest and greatest trends in UX and what have you, but you have to add to your repertoire prompt engineering. You really need to learn how to be good at that. You need to also develop much more resiliency than you had in the past.

Meaning your first prompt is likely not going to get you what you want. Um, so you have to keep trying and you have to keep refining it. Um, so if you don't have that resiliency that, Hey, I'm gonna keep doing this over and over again until I get it right, um, it's gonna be a frustrating experience, but if you go into it knowing you have to do that, it could be really, really rewarding.

And even with all the multiple attempts, you're gonna realize you just saved yourself a ton of time, uh, than ever before as a, as a professional. Yeah. On a very basic personal level of that, I found myself using chat GPT to design a flyer for an upcoming event.

Wow. And I ended up getting sucked into two hours of re-prompt it over and over again till I got just the perfect flyer. But I found it fun.

It was kind of exciting. Yeah, yeah. In a way it's like, um, mentoring, like a junior resource, right?

Um, except mentoring, you're probably being more prescriptive, whereas, you know, in the case of an LLM, you're really more focused on asking it the right questions so that it can perform the work that you want it to. So that's the shift, which is, you know, a little bit different than before. Absolutely.

So humans are gonna need to evolve as this technology evolves, and we see it advancing quite rapidly. So what advice do you have for, um, the future and what do you see the technology being used for in the future? So I think, uh, I'll start with the last question.

I think the technology's gonna be used for rote, mundane, soul sucking tasks, especially when they're at large volumes, right? Because sometimes if you have a soul sucking task, but there's only one, it may just be faster for the human to just go ahead and do that. But if you have large volumes of that type of work, for example, automated testing in the world that I live in, that's something that I think, um, is going to be shifting more and more to agents.

Um, as far as, um, you know, what does this mean for, you know, humans? Um, I think transparently there's one that I have some pretty clear views and perspectives on, and one that I'm currently trying to work on. Um, so if you're an experienced professional, you know, senior, I, I think your responsibility moves even further away from being hands-on and more review, um, and training and reinforcing what you want the agent to do.

So the next time it takes a run, it produces better output. Okay. I think the biggest challenge we all as an industry, um, have got to figure out is you don't get to be a senior, a lead, et cetera, without having done the junior level work.

The trouble is agents can do all the age, the junior level work now, right? So you have really the, the, the agents competing, I think with the juniors who are coming in straight outta school trying to learn their craft, how, how do we as an industry ensure that they get the proper growth and support and training and experiences and at bats so that they can eventually become the seniors to oversee the agents. And I wish I could say like, I've got this magic, you know, formula.

Here's the, you know, 10 steps you need to go through. But you know, in the consulting industry, right? Part of our, um, lifecycle is bringing in juniors, growing them, right, having them advance in their career so they can move on or, you know, do bigger and better, have more responsibility.

So that motion now is really dependent on, so now with the gen agents coming into the mix, how do they coexist? And there's no clear cut answer right now. And that's something that me and my leadership team are constantly talking about is how do we still grow and groom our junior level resources while we have agents that can come in and clearly do this faster and cheaper.

Um, so that's the nut that we need to crack in. That's a work in progress. Yeah, absolutely.

Um, definitely something to think about. Well, if there was one key takeaway you could leave our audience with today, what would that be? I think, um, like the internet, like the cloud, and now we're talking about generative ai.

Um, one AI has been around for 50 years. Like, I don't think a lot of people realize that. I think, uh, the promise of AI has been hampered by the fact that you really needed to be pretty techie to really leverage it.

Um, and with generative ai, it's now just opened up the power of AI to be harnessed by people who don't necessarily have to hide, have that highly technical skillset and a data science science background. Um, so embrace it. This is a great opportunity.

It's a great equalizer. You don't have to be the smartest data science nerd on the planet or understand neural networks to the nth team degree, right? And how it all works underneath the hood, you can get the benefit out of it and start using AI to solve, to produce solutions, which is what I think has been missing.

And so for those companies that are trying to decide whether we jump in now or later, jump in now, right? You can dip your toe in the water. You don't have to jump in full bo but you gotta dip your toe in the water, operate with, you know, the upside in mine, I know there's always fear and uncertainty, right?

The internet could have been abused, it's still being abused. Uh, the cloud can be, um, you know, abused. It's still being abused.

But by and large, it's been a useful ad to our lives. And I think gen AI is the same way. Um, so I, I think we can learn from recent technology trends and say, Hey, what could we have done better?

What do we wish? Like, I think if you go back and you talk to the late adopters of cloud, they would've said, I wish we studied five to seven years ago. And so here you are, you've got gen ai, you have that opportunity again to make that same mistake or to learn from that mistake and say, no, we're gonna start experiment with this.

We're gonna embrace it. We're gonna get our people trained up on it. It absolutely, uh, boggles my mind to talk to some clients, and they, they've been told they are not allowed to use chat GPT.

Okay, well, in five years, your competitors who don't have that mandate, they are going to be light years ahead of you now, because the speed at which you can harness results from generative AI is like nothing we've seen in the past. So the, the, the lead that your competitors are gonna build may not be something you can overcome if you have such draconian rules, govern it. Yes.

Understand it, understand the risks, put safeguards and safety rails. Yes. But say, don't use it at all.

I, I would think that's a mistake. So for anybody out there that's listening and you're kind of pondering making those types of policies, or you have those policies in place, I would really ask you guys to reconsider. All right, well, I agree on that.

So thank you so much for coming on our show and sharing your insights with today. It's been great. Awesome.

Thank you. It was really fun. All right.

And thank you to our audience. Stay tuned. There's more.