AI is the E-Bike for the Mind – Updates in Harnessing Generative AI for Operational Excellence | AIE 2024

Hi, welcome to my talk. My name is Joseph Phoenix, and we're gonna be talking today about generative ai, the E-bike for the mind. And more specifically, we're gonna talk about how we can harness generative AI for operational excellence.

A little bit about myself. My name is Josephine again, and I'm the managing director for artificial intelligence and Machine learning for Enterprise Vision Technologies. I also host a weekly countdown to a GI newsletter, um, which you can jo join on LinkedIn.

And what we're doing is we're tracking and keeping track of and counting down to a GI, which is artificial general intelligence, uh, a little bit more about that, um, in the newsletter. So please jump in and subscribe and, and check that out. But for now, I wanna talk about a shared passion that I've had for AI for a while.

Um, back in early, uh, 2022, a buddy of mine, a named Jean Kim, where sitting at an advancing Women for Technologies, um, event. And, uh, gene started to asking me about things I was doing. And I talked to him about generative ai and, uh, that's all we talked about the, the rest of the time.

And he, he shared with me some ideas about what he wanted to do with an idea that he had. And the idea was, was that he had around 1700, uh, videos and, and talks that he wanted to synthesize into retrieval, augmented generation bot. And that's exactly what we did.

And, and he caught the bug. And I caught the bug. And, and since then, it's just, uh, been off to the races with generative ai.

And with that one, things started to progress. We started looking at like, really, where is generative AI at? Well, we talk about at the bottom here, personal generative ai professional generative AI team, generative ai and enterprise generative ai.

But, you know, wait a minute. Why do we call this the e-bike for the mind? Well, let's step back a little bit to the PC era.

You have Steve Jobs in the PC era, and what did he say about the, the pc? He said that there was a study that was done, uh, by the Scientific American, and what they did was they tested different, uh, species that were trying to get from point A to point B. They tested birds and different animals and different things.

And, and what they, they found out was, is that the condor was the most efficient animal to get from point A to point B, and humans were somewhere disappointingly down the bottom, uh, one third of the list. But the, the folks at the scientific American were creative enough to test a human on a bicycle, and the human was twice as as good as, as the condor on getting from point A to point B. So what did that tell us about humans?

Well, it told us that we were innate tool builders and that we would fashion tools that extend our capabilities. And so when we look at the pc, the PC was the bicycle for the mind. And what is generative ai?

Well, generative AI is the e-bike for the mind. It can get us there much further, but I don't know if you've been on some of these e-bikes. Um, they are for real, you know, and if you get on 'em, I don't know if you've seen, um, the discussion with this, uh, picture here.

Well, not a helmet, right? And you've got the, the, the fast e-bike. Well, when you engage that, that, that, that power, you really have to hold on.

And when you engage the brake, you really need to slow down. Well, that's exactly with generative ai. What we don't want to do is we don't want to have an, an incident where we crash and burn with generative ai.

So, well, what's the, the answer there is just make sure that when you have these large enterprises or these organizations and you're trying to gain efficiencies, don't take too much track out of your environment. Don't try to say, I'm gonna make this train. You know, do a 90 degree turn.

It's probably not gonna work out for you. We haven't had the generative ai, um, challenge where things have come completely off the rails, and let's hope that we don't get there. But again, as you're implementing these technologies, remember the e-bike for the mind.

And remember, just be cautious about how much track and automation that you're trying to hand over to the ai. Let's take a little step back about the accelerating developments in ai. This was originally published, um, for me, one of the talks that I did back in, uh, early 2020, late 2023, early 2023.

Um, let's, uh, I'm gonna pause there and I'm gonna go back one slide. Perfect, and 3, 2, 1. Let's talk a little bit about the accelerating developments of ai.

This is from a talk that I did earlier in the year, and this is tracking at the top left, the LLM based autonomous agents, uh, in addition to some agents you see at the top right for hugging GPT, and along the bottom you can see machine learning and AI and the number of papers released from archive. Well, what is archive? Archive is a website that does pre-releasees on, on papers, uh, so that people can see before they get peer reviewed.

Um, the time that it takes to get to peer review is, is a while, but in this scenario, they release 'em for pre-release for people to see. So as you can see on the bottom left hand side, you have machine learning and ai, the number of papers that have been released. And then here in the middle, you see the large language models.

And then, uh, let me do this again, apologies through 3, 2, 1. Now let's talk a little bit about the accelerating developments in ai. As you can see here at the bottom left, we have machine learning and ai.

And you can see from years past that we have a lot of papers that have started increasing in the late, uh, uh, 20 20, 20 21. And then here in the middle, you can see language models themselves increasing in 2022 and 2023. And then you can see here in 2020, late 2023 into 2024, you can see the large language models were progressing.

And up here you can see these large language model autonomous agents, and these agents are also progressing at a, a rapid pace. And then this is one of my favorites from hugging GPT that was released at the time. But what's most important to focus on the timeframes, as you can see here, at the end of 2023, things started rapidly increasing, and they're even increasing faster now, and months and years of, of work and advancements are being done just in the, in the matter of weeks.

Now, if we move forward, this is from original, Google has no moat. And what was kind of the, the w waking moment for this particular situation was the timeframes. Again, you can see when back in March when this was released, that it was 16 days before, uh, they had a model that was fine tuned to compete with, with Google Bard.

Well, also at the time, the tool perplexity came out and it was a closed source tool. And, uh, some open source teams, including ourselves, released some open tooling that was able to, to compete with the perplexities of the world. And so you see these companies spending millions of dollars along the open source community being able to catch up.

Well, where have things progressed since then? Well, things are moving at a, a very rapid pace. And you look at these software engineering agents, and I've got three of 'em here at the top.

Now again, what do these software engineering agents do? As a software developer, you basically have a source code repository. You raise issues, bugs, and features.

And with that, these software engineering agents can actually read through the issues. They can read through the code base, they can stand up test environments, they can then execute these tests on the code, and then they can come back with a solved set of code for you to pull request and to potentially be able to just have an automated system that that can actually take care of the software for you. So in this scenario, we see the software engineering agent out of Princeton.

You can see the Auto Code Rover out of the National University of Singapore and GPT Pilot out of Y Combinator. This is a SWE software engineering benchmark. And this is the leaderboard of this, which we can share a link, uh, later so that you can see this.

3%. And what does that mean? 3% of the software engineering tasks that were thrown at this code agent, it was able to successfully pull the issue, find the items, the classes inside of the software, write the code, test the code, and put in a pull request that could be reviewed and then pulled into the source code.

So this is a, a major advancement, and we think we're, you're gonna see a lot more of this in the future. So, taking a little step back to more of a primer on where we're at in the gen AI and AI landscape, obviously there's these foundation models that everyone's heard about, the GPTs and the llamas of the world. There's a a lot of advancement in retrieval, augmented generation in addition to these tools and AI integrations along with these multi-step planning agents like SWE agent, and then ultimately these enterprise ready agents, which we'll talk about in a moment where we're actually integrating these safely into our enterprises.

So with that, I want to just give a little bit of the landscape that's important for you to be thinking about as you're deploying generative AI in your environment. As you can see here, you have your cloud providers, you have your internal private, um, considerations for private data. Are you going to do this in hybrid cloud?

Are you gonna use closed source, open source, or SaaS models? There's a lot of different orchestration frameworks out here, your vector databases, the things that you're going to do on the edge for inferencing, whether that's hardware or software, and how we're going to test and evaluate these things. So when we look at this ecosystem, here's a number of players.

Obviously our traditional cloud providers, as you can see, Azure lama, which this is LAMA two, LAMA three has just recently come out along with our, our inferencing hardware, which now we've got a lot more inferencing players coming out. And then in the testing, there's still, these are still the main players here, Lang Smith from the Lang Chain Group, along with True Era and, uh, rise Phoenix. So stepping to, uh, the use cases for retrieval augmented generation, these are the main use cases that we're seeing.

We're seeing a lot of conversational AI chatbots where the large language models have language understanding and can have long conversations with the dimensions of interaction. We see question and answering. The large language models can sift through large amounts of data and answer questions.

Semantic app search, meaning that if I want to be able to have support cases or documents or product search or in the sense of security looking for vulnerabilities, a lot of this semantic app search is coming to bear. In addition to research and analysis, research and analysis is, is growing quite a bit because for decision support and for research, a lot of the drudgery can be taken out of this to bring and and surface the citations that can guide you on your decisions, whether that's financial, legal, um, coding, or again, um, security, uh, vulnerabilities. This is a high level of retrieval, augmented generation.

This is more the standard retrieval augmented generation more from a technical perspective. Now, when we look at these coding agents, this is more of, again, a technical perspective from these coding agents. And we talked before about hugging GPT.

This is one from Microsoft that was released. Um, and, and this one's actually progressing quite a bit, but let's look at this pattern for a minute. So we have our user and our users interacting and, and wanting a task to be completed.

So once that user asks for that task to be completed, what happens on the backend? Well, we have these purpose-built agents that can do very specific and precise things in relationships with the, with each other. In this particular scenario, as we talked about for research, well, the user asks the planning agent, I wanna research a specific topic.

Once that happens, we have a search agent that can research, conduct the research, do a web search. We have a curator that looks for the, the proper, um, references. We have an editor that can edit that.

And then we have, we pass that over to the writer. And then that writer along with the critic, um, validates the writing and to make sure that it's formatted for the, with the right tone. Ultimately, that's passed over to a designer and then a publisher that would publish the final research.

And these sort of agents are getting more and more powerful, um, as time progresses. So I want to take a a, a deeper dive too, into these enterprise application integrations. So this is a, uh, a backend, um, pattern that we're seeing a lot of, whether this is on your cloud provider or whether this is on premise.

Um, you are looking at the ability to have integrations with your traditional SQL databases, integrations with your various APIs, in addition to being able to do web search safely through a Google or, or a various web search, um, of your choice and also internal searches if you have those as well. And then of course, our vector stores, which provide the semantic search capabilities for documents. And then in the backend, right, our unstructured data.

And what's important about this is that there was a report that was done a few years back about structured data and non-structured data. And at the time it was an 80 20 split, that that structured data was about 20%, and unstructured data was about 80%. Well, the latest stance on this is that unstructured data is actually getting closer to 90% of our data.

So why is that important? It's important because that's where the value the treasurer in our organizations and what's actually taking place in our organization is found in this unstructured data. So the important thing thing to see here as well is in this particular scenario, this is secured by Azure's, um, open ai, but Azure is now supporting a lot of different large language models along with the other cloud providers are supporting different large language models as well.

But as you can see here, the beauty of these things is organizations like ServiceNow and obviously Microsoft. With with teams, we can take our identity and access management and all the backend connectivity that would be with your Microsoft products and your copilots that you're building in addition to things for ITSM or human resources or things that you're doing in ServiceNow, and integrate those directly in with your agents and have those pulling specifically precisely the databases that you want, the APIs that you want to hit, the specific controlled web searches in addition to the documents that you have that you want these people to be able to access. And why this is important is, is that I can control the role-based access on what's provided from these agents, the toxicity, the efficacy, the ethical nature of it.

I can also make sure that people are not getting access to specific pieces of information that are not, um, they're not authorized to see. And this whole process is, is really moving forward to include these enterprise application integrations. So with that, I want to take another sort of step back and, and give some some risks and insights as we count down to a GI.

So what's happening, as I've said multiple times this space is, is speeding up. It's rapidly changing. Um, there's gonna be more risks of outdated understanding as we continue down this pathway.

Um, and we have to continue to embrace this a, a advancement and prepare for it. When we look at, uh, a GI or artificial general intelligence, it is gonna be capable to do, um, the 90th and 95th percentile of, of many human tasks. So we need to prepare for that and we need our educators, um, along with, you know, our government entities to think about how we can have, you know, people focusing on the tasks that are gonna be difficult for the AI to actually accomplish.

And making sure that our people are trained to do those things so that we can progress along with the ai, as I've said multiple times. Also treat your data as treasure. It's, it's, it's never been more, uh, important to, to look at treating your data as treasure, but also how do you take that raw information and transfer that data into actual knowledge and then gain value of it in your organization.

We also have to be constantly concerned about our, our ethical components and our ethical considerations. And this is another thing for education. When we look at some of the leaders that are talking about what should be trained and what our, our curriculum should look like, um, one of the things that's emerging is, is that that, that the curriculum needs to be a little bit wider.

You can't only focus on, let's say the software, uh, discipline and the engineering discipline. We need to have, uh, a component of, of the ethical piece and the impacts of what these algorithms are going to have. And so as we broaden our curriculum, the people that are developing these algorithms, they're workloads in, in school and curriculum, are going to be expanded to understand the downstream impacts that they're having, not just on the technology but on society as a whole.

Um, and I've been saying this for a while, prepare for your expanded personal devices before we were talking about the pendants and the humane pins and meta and this, uh, the rabbit ai, well, just recently meta announced their open sourcing their os. Um, and what that's gonna mean is that, that pretty much any manufacturer, hardware manufacturer will be able to have an AI device that is connected in with video and audio and voice controls. So we, we need to prepare for these things, especially, um, in your organizations.

And so with that, I'd like to take a minute to, again, thank you for joining, um, our talk here today. Again, my name's Joseph Phoenix, uh, from EVT and I, uh, really appreciate the time that you've taken, um, to listen to our harnessing generative AI for operational excellence.