Navigating the AI Frontier: Harnessing Generative AI for Operational Excellence | AI in Action 2023

Tray table's up, seat backs in their upright and locked position. We're going to jump into the frontiers of generative ai. So, uh, I wanna share a, a little bit of story before we get started with a good friend of mine named Jean Kim, who I've known for over a decade.

Earlier in the year, Jean and I were at a, a charity event for advancing women technology, and we sat down for lunch afterwards and talked about some of the things that we had been working on in generative ai, and that's all we spoke about. And Gene started sharing with me some things that he had been wanting to accomplish for months about, uh, generative ai and with the talks that he had been doing over the years for DevOps and enterprise DevOps. We'll talk a little bit more about that later.

Um, but I just wanted to start by sharing out that, that small story of how this, this journey began. I can't stress to you enough the growing wave of enthusiasm with AI and generative ai. In the top left hand side, you can see recent, uh, mentions and earning calls have spiked the investments from top players, Amazon, Google, Microsoft, Nvidia, are, are, are fanning out across the industry.

And the Google searches from generative AI and AI in general are, are up and to the right as we look at the technology increases. org. org and start reading these papers.

What they are is they're pre-released peer review articles. But what I am saying is, if you're embarking on a generative AI project, someone on your team needs to be keeping up to date with what's going on in generative ai. Now, starting at the bottom left hand side, you can see machine learning and ai, and you can see these advancements in publications by year from the 1990s on up into 2021.

Now, if we look at the, the language models in general themselves, you can see that those have been increasing. And then on the right hand side, obviously these large language models, again, up into an an exponential upward pointing curve. Now, I want to also point out not only the advancements in these large language models, but also in these autonomous agents.

Now, you can see here in the top, these agents and agent frameworks, which we will talk about in more in depth, have had a, a, a, an increase in their, you know, their speed. And with the advancements that are being made with these, um, generative AI agents. One of the, um, papers that is really exciting to me, um, is this, uh, hugging GPT paper.

Now, if you can see here in the center, this is about using a, a large language model for a planning agent. And planning is one of the areas that we, we really do need to focus on longer term. But once we have these planning agents and tasks set up, then we can have these smaller, more purpose-built, large language models or agents or chains of agents actually executing the tasks for the large language model, the primary large language models to synthesize and, and validate.

Moving forward with that, I want to actually again, focus on these dates. If you notice, decades of research are being done in months. Now when I talk about these archive papers, these are not, uh, just tweets.

These are papers that take months to write. The teams are growing exponentially in size, um, and the advancements are, again, moving at a very rapid pace. I wanna again, step back to again, some of these advancements and how the major players in the world are really taking notice.

This is a, a leaked memo from Google earlier in the year, and they were very, very concerned when this memo came out. The memo was validated, Demis bu A CEO of Google. DeepMind validated it.

He didn't necessarily agree with all the findings from the research, but he did, uh, uh, uh, nevertheless validate it. I think for, for sake of our conversation, what I want to be able to point out is the dates. Now, if we look at some of the dates on here, we have March 3rd, we have this, uh, llama model that was, that was leaked out.

It was originally meant to be for researchers, and then on March 3rd, it was leaked to the wider community. This has since gone to LAMA two, et cetera. But at this time, on March 3rd, you can see here this model was leaked.

Now up here on the chart, within just a few weeks, the open source community, and it is, it is, it is, you know, slated here from March 3rd to March 19th, that in 16 days that the open source community was able to take this 7 billion parameter, 13 billion parameter model here, and fine tune it so that it had parody with Google Bard. It had been a ton of investment from, from Google at the time. So this, uh, raised their awareness about the power of the open source community.

I want to focus your attention down here to the bottom right, which I think is one of the things that we're really going to see advancing in the future. And this is sort of transfer learning and fine tuning. This transfer learning and fine tuning is where we can take a foundation model and we can really take a set of tasks or subtasks that we want it to focus in, and we can fine tune its capabilities so that it can do those tasks very well.

Now, maybe it can't do it as good as a GPT-4, but again, when we look at cost savings, and we will look at token costs, which we'll review in a bit, being able to fine tune our own internal models is something that again, I think is very exciting. As you can see here, this LAMA adapter, which was highlighted in this leak as well, they're fine tuning these models in as little as an hour. This remind me of, uh, early days of DevOps when we were talking about releases per minute.

Now we're having, uh, you know, releases per day or hour for these large language models as well. So another thing that, uh, uh, my team and I had, uh, looked at at the time was, was this, again, this agent concept. And on the left hand side, this is an organization called Perplexity ai, which is one of my favorite tools.

Um, these large language models called do what they call hallucinate. So what we do is we can go to a, an organization like Perplexity and they will look up and do an internet search on your question, and then they'll do a retrieval augmented generation where they'll bring that data back, synthesize it, and validate what the large language model is saying. And you can see here on the left hand side perplexity on the right hand side, this is an, an upstream project called GPT Researcher.

Now, our team, our team has gone and taken this open source, uh, code and, and, you know, with about, uh, 48 hours of elbow grease, we were able to, to take and do a critical analysis, as you can see of that leaked, uh, Google article. So again, on the left we have perplexity with millions of dollars in investment, and on the right we have open source projects and, and our teams with a little bit of elbow grease and 48 hours that can get very, very similar results. And this is very, very exciting for, um, you know, not just the major players, but for regular, um, practitioners like myself.

I wanna take, again, a little bit closer look at this emergent agent architecture, and you'll, you'll hear a lot about agents and agents swarms, and I want to kind of demystify that very quickly. What are we talking about with agents and or what GPT, uh, OpenAI and GPT is calling them? GPTs, you can see here these, uh, single path agents now through prompting and some chaining, we can say, here you are an expert at searching for documents, right?

Or you are an expert at Power bi, or you are an expert at sql. Then along with some coding in the background and some magic, we can make it so that that individual agent can have a very specific and precise job that they do. And then as you can see here in the center, you're an agent that's an expert at validating data, a validation.

Now, once we have these single path agents in our value stream, we can take those single path agents and have a MULTIPATH agent that you can see here with this multipath agent that is the agent that we saw before could do planning. Now we have an end user that asks to execute some sort of steps. The multi-path agent knows it has access to these other single path agents.

It executes the plan, initiates the work, then synthesizes the work, validates the work, and gives it back to the user. At the top here is a, is a paper from Microsoft Research. Let's take a closer look at this.

Now this paper is called Auto gen, and um, you should definitely go look at auto gen. It's got a lot of excitement. Um, Microsoft has invested a lot and has given this back to the community.

And one of the things at the bottom left, this particular agent, um, a four agent where we have our commander and our rider and our safeguard, this one has, has garnered a lot of attention for code creation because what's happening here is an end user is asking the commander, again, the multi-path agent, Hey, I want to create this bit of software, these set of functions with these outcomes. What happens then is it passes that on to the writer agent, and then the writer agent writes the code, and then it passes its code over to the safeguard agent, which then validates that the code is safe and accurate. And if it's not safe and accurate, then the safeguard agent will tell the writer, rewrite this code.

This can go on and on and on. But the beauty of this is with the right scope and with the right prompting, we've seen a very high degree of fidelity in this sort of agent framework. And, and we as a team are, are investing in this as well.

But very exciting, and I wanted to share that with the groups. Another thing here is, again, this, this concept of frontier models versus our open source models. On the right hand side, I want you to pay close attention and orange to these frontier models, the closed ones, it's sort of a bell curve, a bell curve if you will.

And, but look in blue to these open source models, you can see that the open source community is releasing a tremendous amount of these foundation models. And this is, um, dated as a, you know, maybe a, a few weeks ago. And there's a lot more open source models and a few, uh, frontier models that have been released.

But 2023 is becoming the year, or has become the year of the open source foundation model. Now, looking again here to the left, we can't count out these major players. Now you can see down at the bottom with Ajax GPT from Apple, you can obviously see GPT five.

We don't know exactly when GPT five is gonna be coming out at the top though, I really want you to pay close attention to Gemini with Google and Google Deep Mind. Now, this model is going to be trained on 65 trillion tokens, and we'll have 8 billion or 8 trillion, excuse me, with a t um, estimated parameters. Now this is very interesting because this is something akin to every word that's ever been written, um, in every language.

Um, you know, as long as humanity has existed somewhere close to that maybe is a margin of error there. But, um, some believe that this Gemini will be indistinguishable from magic when it comes out. I'm excited to see that and I hope you are as well.

So then also taking a look at, um, a HuggingFace, which we'll talk about in depth in a few minutes, but, but HuggingFace is really, I want you to think of it as the GitHub for ai. It keeps the open source leaderboard, as you can see here. It also keeps the, um, chatbot arena where the open source and the closed source frontier models will battle it out in these, um, validation, uh, tests to see which one is more highly accurate.

Taking a little closer look at the open source LLM leaderboard. These really come in three major categories, and it's really about the size and the number of, uh, parameters in their capabilities. You have the larger than 30 billion parameter groups.

And, and right now just, and this changes daily, so it may have changed by the time you're watching this video, but the leaderboard here is a fine tuned tiger research model at 70 billion parameters. But, but notice that underneath this is a 34 billion parameter model, the e model. Now, what's very interesting about this model is that a 34 billion parameter model, many folks believe that this is a sweet spot because a 34 billion parameter model can run on and potentially be trained on your traditional, let's say a 24 gig GPU, uh, commercial off the shelf GPU.

When you get into the 70 billion, these are very expensive pieces of hardware, potentially what they call gold plated hardware with InfiniBand and things of that nature, which really put them out of reach from, from many researchers. So this is an exciting space to see that these smaller models are, are competing with these larger models. And we believe this is a trend that will continue a little bit lower, uh, into the, the size of parameters is the 13 billion, 14 billion parameter models.

And the Quinn model is one that you should definitely take a close look at. And then down here at the lower end is the 6 billion and 7 billion parameter models. And you can see again here the ye uh, Quinn model.

In addition to the menstrual model, a lot of people are excited about minstrel and they have announced that they will be releasing, um, some larger 13 billion models, and hopefully we see a 34 billion, uh, parameter model come out of the Mytral group. Um, I wanna take a minute to talk about navigating this landscape. If you haven't had a chance to spend hundreds of hours doing experiments and looking all the, at these tools like I've had the pleasure of doing, I just want to throw out a few terms.

Um, you're going to see these single large language models and these ensemble mixture of models and retrieval, augmented generation. This is a paper by Jeffrey Hinton, you may have heard about him from Google. And you can see this is this, uh, mixture of experts, which GPT-4.

Um, we, we now know is what they call a mixture of experts model, where it has multiple models that have been trained on various specific things like legal or like medical or accounting and things of that nature. And then when you ask it a question, that model that's been fine tuned on that particular thing is the one that's responding back to you. And you can see here orchestration vector databases in testing evaluation along with the foundation models and cloud providers that we spoke about.

Now, taking a look here, um, this is an ecosystem. I won't go through all of these, um, but you can see here in the vendor ecosystem for vector stores, pine cones of the world, you can see here, um, mosaic and Databricks, very, very important players in the space along with cloud provider partnerships with anthropic and cohere and obviously OpenAI, Azure, OpenAI, along with the LAMA and foundational models. I want you to take a look here as well.

I notice that, uh, the M twos are over here from Apple. It's interesting because for inferencing, um, the VM twos are actually becoming a very cost-effective way to actually do inferencing with Lama CPP. If you haven't seen that, take a look at that.

And other players that are emerging in the space are these sort of end-to-end integrated offerings like tta, where the extraction, encoding, indexing, retrieval, re-ranking and summarization are all done with EPI endpoints. I i, I recommend you take a closer look at them if you're planning on buying versus building. So now we're gonna take a step back and do a little bit under the hood, uh, look at things.

So we have these, um, few things called tokens, embeddings, and transformers. I want to demystify this and simplify it as much as I can. I want you to think of these as word puzzles or image puzzles and language understanding.

As you can see here, the tokens and the encodings, think of these as just words, but the individual puzzle pieces and these unique representation of these puzzle pieces. Then think of embeddings as the relationship between these puzzle pieces. What comes first?

What comes next, and how are they related to each other? Positional or relevance? And now I want you to think as the, the transformers as really they are taking these puzzle pieces and these puzzles that they've seen millions and millions of or even billions of.

And they're taking those and synthesizing those into language understanding. And I want to take a closer look underneath the hood for these transformer architecture. As you can see again here on the left hand side, we have the sentences and the relationships associated with the sentences.

This is known as the self attention blocks. So as these models are trained on the self attention, the more they see of this data, um, the, again, the more aware they are of these relationships. I think one of the most important things here is that transformers, they may not be the future for, but right now they have become what is called a, a unifying architecture.

So they don't only see text, they can also help us fine tune and train on audio. Um, as you can see at the bottom here, this is a, an audio that has been transformed into an image, a spectrograph, and you can see video and images. So this transformer architecture is bringing the communities together for the fine tuning and training of these models, which is exciting.

Now I want to again, take a step back to the, the use case that Gene and I were working on. So as you can see, the, the problem that Gene had is Gene has over a thousand of these videos that folks have done. And Gene has, has watched and seen many of these.

But one of the things he mentioned to me is like, Hey, I can't necessarily take the time to watch every one of these, but what if we could summarize them? I could read a one page summary of them and make it so that I can then level up the talks that people were most excited about to get them on the plenary stage at the, um, these DevOps events. So we took that challenge to heart.

And what did we do? We actually started building around, uh, the token limit for some of these large language models. And we were building retrieval, augmented generation, and doing some techniques to be able to synthesize those chunks of data, um, so that we didn't, uh, overcome the token window.

5 turbo came out with a 16 K context window and many of those talks just fit right inside the window. So our retrieval augmented generation for this particular use case, it was not necessarily needed, which was exciting to us. Um, 'cause we were like, yep.

Um, you know, like we said, things are moving very, very fast. But the outcome of that nevertheless, was these talks and these summarizations and you can see here, um, and you can go to the, um, site and see the HSBC talk and the summaries associated with those talks. Very fun time.

Um, there's also with Jean's latest book that you might check out, um, I recommend that you do wiring the winning organization. We have some synthesis of those, um, components of that book that you can see integrated in some of those, those use cases that have been, uh, created for DevOps. I also want you to take, um, a minute to, to think about strategic decisions in your organization and some, I'm gonna provide some critical resources.

This is the AI Infrastructure Alignment Alliance. They have a very extensive list of all of the players, um, that are involved in the things that you should consider. On the right hand side, this is primarily this right hand end is the area that we're focusing on more for generative ai.

As you can see here, the foundation models, the vector databases, and the agent frameworks. Um, if you want to get a deeper dive of this, please reach out to me, um, and we can get you in touch with the right folks from the AI Infrastructure Alignment Alliance and, and talk more in depth about this. I want to give you some additional cre critical resources.

Obviously your cloud providers HuggingFace, again, because this is where all of the models, you can actually download the models and use the models, and it's a center of gravity. Uh, lang chain is another tool that's an orchestration that you really should take a look at. org, like I mentioned, and there's a couple of sites on YouTube that, that provide updated videos.

So that's our easy ways to keep updated. Another thing to take a look at is the risks and insights. You know, as you can see here, this is a rapidly changing landscape.

There's dangers of outdated understanding, and you should try to embrace this, uh, advancement. One of the things that you will hear me continually say, and I don't think I will stop saying this, is, is treat your data as treasure. A recent study by a Luther AI showed that 92% of training data is not used 99% of the time.

So that percentage that is, is very, very valuable. And as these models, the training gets smaller and smaller. Why is that?

It's because the fidelity of the data, the quality of the data, we're not using compute to compensate for the, the issues that we have inside of our data. So treat your data as treasure. It's becoming more and more important.

I, again, I can't state enough about these ethical components. It's very important to consider the ethical components. If you're doing ai, be very mindful of the ethical considerations, um, and take a look at and prepare for expanded personal device use.

Now I'm just, just talking about, you know, cell phones and things of that nature. You can see the humane AI pin, you can see Meta and Ray Band, and you can see the pendant from we from Rewind. These personal devices are coming out and, um, you really have to prepare for them as an organization and their use and their capabilities.

Um, I wanna propose, um, as we're slowly coming to a close here, these ideation symposiums inside of your organization and bringing in professionals as part of these symposiums to talk about customization. And this is talking about, let's say in consumer products and CPG, the next generation, let's say, in retail experiences, enhancing marketing and sales, customer care and improvement, which is very important. Um, product development.

And again, when we look at consider considerations for your enterprise, understanding the differences between content, fact and inferencing, validating these things, these things hallucinate it's important. Security, as you can see here, they have a lot of these attacks and adversarial attacks that are coming about and making sure that people, uh, when you give them access to a large language model, that the data for retrieval, augmented generation has role-based access controls with it so that you can audit these things and you don't wanna give access to various pieces and parts of sensitive data. Like PII, very important.

So, as I, in summary, I wanna, again, thank everybody for listening to this talk. We've had a lot of talks here about foundational models, about some of the retrieval, augmented generation components, about these single path agents and multi-step task agents. And ultimately, this pathway is leading us to these hyper-personalized agents that are going to be very exciting.

Um, but again, we have to navigate these things and bring 'em to our communities and users safely. And with that, I wanna thank everybody and this is a, a midjourney prompt that I use to generate this. And you could see at the bottom it's a, a whimsical illustration of a spaceship launching into the unknown, which I think we are, we're going into this quest, which, you know, in a journey you kind of know where the outcome is.

But in a quest, which we're all on together, we don't know where it is, but it's very exciting. And again, I want to thank you for joining me.