Operationalizing AI: Building and Running Generative AI Applications at Scale | DevOps Experience 2023

Hey, everybody. Uh, welcome to our session. We're gonna be talking about, uh, operationalizing ai, right?

How do you make the most out of what's, uh, what's out there? How do you get started as an organization? All those good, uh, good topics.

Uh, my name is Damon Edwards. I'm one of the, uh, product leaders at, uh, PagerDuty. I build, uh, uh, focusing on, uh, AI and automation products.

And, uh, I'd like to introduce, uh, start with you, John. I, right here, John Willis. John, tell us who you, you're, Uh, I'm John Willis.

I'm also known as Bache Galu. Um, yeah, just right now, uh, independent, working with a couple of clients, tech, strong, open context, and, uh, soon to be Mongo. Um, uh, just exploring a lot, uh, this new frontier called Jenner Ai.

Got it. And, uh, to my left here, the esteemed Patrick Deis. How you doing, buddy?

I'm good. Yeah, I guess, uh, for me the introduction would be, most people kind of know me from DevOps and the DevOps handbook, but recently I being like a very enthusiast on the new field on gen ai. Um, so I guess that's where we're here.

Yeah. I think you guys are selling yourself short to the, uh, the deepest diving folks I know in this, uh, in this area, getting your hands dirty, helping companies do this. So, uh, the modesty doesn't help the panel guys, so gotta sell it better next time.

That's All we have here, Damon. That's all We have. Yeah.

So, alright, so let's get started. So, uh, you know, I, I like to say to people like, look, most companies, you're not building ai, right? You're operationalizing it.

You're trying to make something, something out of it, right? Um, so, you know, uh, let's start with you Patrick. Like, if you're, you know, gonna talk to, to someone, they're saying, Hey, how do I get my company started here?

Like, this is massive, you know, landscape and stack that can worry about, you know, I'm not sure how to get my team moving. Like, where would you tell them to start the focus and, you know, maybe who would you get focused or how would you get focused? Just give us your thoughts on, you know, how would you advise a company on, on getting started?

Yeah, I think that's, um, a very timely question in that, um, the way that I would want companies to be excited is probably different on how they get to be excited. And I'll, I'll see the difference is that I'll, I'll love to have them kind of think about their use case first and think about how valuable this is gonna be, um, and then have, start creating trials and whether that's with your, uh, chat, G P T and so on. But the reality is probably they're gonna be pushed by our competitors are doing this, and then all of a sudden there's a rush, uh, kind of from marketing that says like, oh, we need to do something.

We need to have a nice, uh, story on this. And then there's the, then there's kind of the scramble begin, like w what we do. So the, instead of thinking the use case, they think like, oh, what can we make shiny new that we can deliver?

So that's already a difference in, in kind of both directions. I hope they will come in with the use case and kind of, usually they're start by tinkering, uh, and looking around what they can do with the new technology. Um, and one of the teams, uh, picks this up if it's a bigger company that might be the data team, because that feels for a lot of companies, the closest team that has the experience with all that AI stuff, and, and they go there.

Um, but eventually it seems that people are moving to the engineering. So that's kind of, uh, the journey that, that will start there. So is that how they get excited?

Um, it's, it's gonna be the, the cool shiny thing they tried either from the chat or you kind of demoing from their use case. Um, and that usually, you know, gets people over the first hump, right? We should look into this.

There's a whole longer journey, but, um, I'll let John chime in here. So, Yeah, thanks Patrick. Um, the, yeah, I think, so earlier this year I wrote an article called, uh, the Rise of Shadow ai, right?

And, and the, and I think, um, there's, there could be interesting parallels between what happened with cloud computing, although I, I think this is a bigger, bigger frontier than even cloud computing is. In fact, we, we were at, uh, Damon, you and I were at a pine cone, uh, uh, conference where they said the last major platform initiative this big was the internet. And, and I, I believe that, but the, the parallel to, um, cloud computing is, um, the IT departments and they didn't know what to do with it.

First it was like, ignore, then it was sort of lock it down. And then, and then, you know, I think many people will tell you that we're 10 years in and we're still trying to recover from people pulling their credit cards out, usage patterns not being controlled. So I, the, the, um, the main point of that article I wrote is really, I believe that CIOs are going to have to treat this.

It seems almost like, uh, stupid. They even have to say this, that this is an IT initiative. Don't let any, uh, part of your organization convince you that this is some other, oh, this is data, this is ai, it's different.

Let's get a new c ai, chief AI officer. You know, I mean, this has gotta be controlled by it because un fundamentally, it's gonna have all the parallels of IT disciplines. You're gonna have to, you, you're gonna wanna control it.

You want to give it the freedom that, that you, you know, ultimately what, what we found out in the cloud computing was we went through these stages, and ultimately it was, once we got a handle on it, you know, most organizations then became a power initiative within the organization. And so I think it's very similar. It's, it's okay to be cautious and we'll talk about the risks, and there are risks, but, but like to Patrick's point, start an initiative.

Get involved in it. Start learning how to use it, don't ignore it, and realize that you're gonna have to kick the tires on every aspect of this stuff to try to understand it. 'cause you will own it.

You know, you'll own it from a breach standpoint. You'll own it from a, um, uh, an operational cost perspective. And more importantly, just, uh, general op, you know, the operationalization costs of all of it.

Yeah. I think you guys, uh, described a pattern that many our listeners probably, uh, felt, right? Which is the marketing driven, like, we're behind, we gotta do something.

So it gets people moving and then they go, well, data science, they know how to do this. So then it's like this data science team suddenly becomes like the hub or the fulcrum for all these delivery efforts. And, um, I've heard from multiple folks, it's like, it just causes, it's unfair to them, right?

It's like you're asking a team that by and large has never really had to be in the mainstream of delivery. Or if they are, they're sort of invited guests, right? And now everyone's thrusting all this pressure onto them, right?

And it becomes this big bottleneck, and then they can only really serve maybe one of those outward facing product teams. And, you know, everybody else gets, gets, uh, gets shortchanged, right? But they still have that excitement and mandate to deal with, and then, you know, they end up going on going their own way.

Right? And, you know, I said that about the, to folks like, hey, like the data science team knows as much about DevOps and delivery as you do about ai, right? So, you know, a marriage is great there, right?

Getting, and I kind of said, Hey, for, you know, this is a great play for platform teams, right? Like, hey, build that dial tone for, you know, partner with your data science folks, build our data engineering, we wanna call it build that dial tone platform so people can focus on higher level, uh, higher level things. Right?

Um, question about that. So, you know, talking about higher level things, right? So once people, you've seen people get excited, gotta do something, right?

Um, what do you, what are the use cases that you're seeing out there either from, you know, personal experience or from, you know, talking, I know we all talked to a lot of folks in the industry. You know, I sort of looked at like, there's the table stakes use cases, right? Then there's maybe, ooh, advanced kind of harder use cases, and then maybe stuff that's still in this sort of experimental, don't bet on it working kind of phase.

Any, uh, any thoughts there kind of on the use cases? What's table stakes? What's, you know, more advanced?

Um, you know, John, what's, uh, start with you? Sure. You know, I, I think, um, the obvious is productivity, right?

And that's either, um, you know, in all areas, but clearly coding, right? I, you know, I, I've, uh, I've been 40 plus years in this, and I've had my runs where I code and then I don't code for years at a time. And I would say since the, um, you know, copilot and chat GT's ability to generate code, I've coded more in the last six months than I've coded in the last 10 years.

And really productive. Um, so clearly, um, the productivity gains. Now, again, like everything else, these aren't magic, you know, you know, there's no magic here.

You have to sort of learn how to, how to use these as productivity age. You don't just replace coders. You, you still have.

But there's so many interesting ways that just from a coding perspective, you know, the, the little snippets or how do I do this? The, the time savings of just understanding how to, how to scrape a, a, you know, a, a parameter list or scrape a, you know, or something like a j ss o n object or, or a screen or all these things. It's just, you know, things that the normal, you know, people who don't code normally now can code more, which then, you know, what you say all, all boats, you know, uh, rise from a rising tide, right?

Yeah. Um, so that's one area coding, you know, again, we can talk about the dangers and the be wears and the cautions. Uh, but, but coding productivity is a given.

Language productivity is, is, you know, reports, generating reports, uh, enhancing reports. Um, you know, we were at just a DevOps enterprise Summit last week, right? And, you know, there's some key executives that are already there.

You know, it's funny, we'd be sitting around a table with some people and they'd be saying, well, you know, maybe I should play with this, you know, this chat G P T or gene AI stuff, as if they knew nothing about it. And then, then they'd be like, oh yeah, I've been, I've got this thing now where it answers all my emails and it's already sorting my email. So, so they're acting like they know nothing about it, and they're already got like, some advanced solutions for themselves, you know?

Um, so there's just, um, just, you know, unlimited amount of ways you can sort of create productivity around anything with language, you know, emails, reports, I mean, I go on and on. And so those are the first two obvious places, you know, just, or one obvious place is productivity. But then I think, um, the things I get excited about is like, how can we sort of use incident management?

How can we enhance, um, risk control? And, and, and we're already seeing, like, if you just follow LinkedIn, the amount of, um, you know, people who are talking about using AI for risk and governance and, and not just products. And that's one last thing I, I wanna say is I, I think right now is not a time to go out and just buy every product that hits your desk.

It's the, like I said earlier, it's the time for an organization to learn how to use this technology, how to absorb it. And then you can make intelligent decisions on whether you should buy or build. But I think you need to build first.

And it's not even build, there isn't really a whole lot of build here. You know, the, the, the mechanics of this stuff is pretty simple. Yeah.

I, I understand. I mean, that's an interesting point, which is like, this is all, there are no experts here, right? This is all brand new, right?

And this is in the last, you know, less than a year has this been sort of ready for, you know, for time, right? Like it is, like it is now. So I think that's an important point, which is getting the use cases from your organization.

'cause I see people like, oh, we're gonna make an AI committee. It's like, well, who's on the AI committee? Well, the people that were smart in the old paradigm, right?

It's like, well, this is the whole new paradigm. Like, you don't know who your stars are. You don't know really what ideas are gonna bubble up.

Um, you know, from, you said you look at productivity and you look at what is the key things that we can do that we can do here. So I think that idea of, you know, getting the organization trained as much as possible so that they become, I mean, I don't know of a better term, AI and native, right? Or something like that, where they start, please Don't, don't get that one started.

Sorry, I won't get that one. That's gonna be, that's very bad. But you know what I mean?

It's like, you need to get them feel like, Hey, I under just like the web, right? People forget it was a new paradigm. It was like, how would I do this thing a webpage and CGIs and wow, you can answer, like, you know, you just would, you had to play around a bunch.

But once you played around a bunch, it sort of became second nature. And now you think about the whole world today in terms of like, you know, what am I gonna do for a webpage? And they start, the new paradigm shift was a mobile phone.

Now you think about what am I gonna do with their mobile phone? So I think, you know, it's a great point about you gotta teach people from the ground up. So your entire organization becomes, you know, sort of those AI experts that you, uh, that you need.

But we're talking about use cases, Patrick, you got, uh, what's, what are sort of the maybe table stakes or advanced use cases you, you see people kind of tackling first? Um, so I, I make the distinction between the ones that enable the business and the ones that enable it. So like John was saying, developer productivity could be one.

It could be generating synthetic data. It could be kind of, you know, various, even the design could be generated much faster. So we don't need the designer, the storyboards getting created.

So that's one, you know, big, big area that has a lot of potential, uh, all the way up to security and stuff. And then there's the business cases where I see increasingly, like you say, the text, email, uh, summarization, translation, uh, kind of those stuff. I do feel then coming back to your point, like, is there, are they table stakes?

Maybe right now there are table stakes, but I, I do think that this easier level that we're building right now, and everybody's scrambling with, and we don't know yet, that's gonna be built into a product Microsoft is ComCom with, you know, there are pieces on email. We're not gonna build that on ourself kind of whole summarization and uploading documents and data that that could be an aw w s one that we're not doing. So we're in this limbo state, like you say, everybody's figuring it out.

But, uh, I, I call it the innovation tax. We're paying it right now in, if you want to be hip, kind of, you, you kind of have to do the dance. Uh, and that's, that's after that, I think there's gonna be a new layer of things like, oh, okay, what are you very specific to your use case?

Like, creating the email is, you know, the easy thing, but why is that? Like what flavor are you gonna put in that like, makes you market leader in your space? And then it becomes like very specific, uh, kind of, um, um, and that becomes more of a table stakes than the obvious one.

Like yes, a transcription. Okay. You know, right now we're being transcribed already, right?

So it's, it's, it's already there. Um, yeah. And that, that is gonna be a lot harder for companies.

Um, so that's one thing that we're leveling up the specific use case. Uh, and then the other part is that we're increasingly, um, having to look for, um, kind of value that we're bringing, because again, there's a lot of use cases, but, um, the unlocking versus the cost, and I'll, I, I always keep bringing that in after the first excitement that we brought it. Like, and we, we got it.

Like, start looking at your budget, right? And of course, you know, the infrastructure cost will go down, but have another thing you need to take care, uh, of, um, kind of, I know it's a use case, but is it like, affordable because people think about this can unlock everything they, they couldn't do before. Uh, but it's, it's not everything there.

So focus in your domain and then within the boundaries that you can solve, because that, that is the other area that I think is, uh, a sweet spot. Uh, and that for that you have to bring in off build business value to match kind of the cost that you're doing with this. Yeah.

That makes a lot, that makes a lot of, that makes a lot of sense. I know, I, um, from my own experience, right? I, I feel like we're at this stage where you, like you said, it's, you know, it's translation or, you know, summarization, um, you know, taking content and filling out a form, right?

Or, or transforming into some other format, you know, generating code or something with a specification, right? If those are all kind of per building, I think the co-pilots, you're right, co it's interesting, I think in the last couple months, really, people were like on the, well, we're cautiously trying, you know, these different coding co-pilots and these types of things. And now it, it's just kind of like, oh yeah, we're doing that, right?

It's just that we just blew right past that. And it's just like, we're not even talking about that. 'cause it's just everyone should have this in their I d E and be working on it.

Um, and then it's interesting, it's kind of like the task level now, right? It's like, generate this, you know, like push a button does something for me, right? And then, you know, I think the really exciting part that I feel like is not yet ready for primetime is, you know, true agents, right?

I don't mean the chat agent that looks at your certain documents, but I mean, like the, you know, we can have autonomous agents that we give them goals and they go and do things, right? And I think why it's so important to learn at that levels, you, you guys were describing right now is like, you know, we have to get that level of capability because as soon as people can jump to the next level and have reliable autonomous, you know, agents that you give a a goal to, and it knows how to use other agents and tools and drive towards a task, right? And do it in a production production reliable way, you know, that's gonna unlock a whole nother level of productivity, a whole nother level of, of, of products.

And John, I think to your point, if your company hasn't built that base level capability now, it's just gonna seem even more foreign, and you're gonna be even more behind, um, when, you know, we hit that next milestone, it's gotta be coming soon, right? I, I think about a story a lot. The, the, the, I think I al always through the years as we go into new technologies, you know, it's gonna sound weird, but I I think it, it applies when I was first selling chef, um, national Institute Instruments was, you know, Ernest Mu, James Wickett, you know, good, good DevOps, um, guardians, the, the Texas guardians of DevOps, if you'll, um, Say that, by the way, I'd appreciate it.

Yeah. But I remember, um, going to Earnest and he was running a project where they built the big sa big pass. It was very early on before people were even using the word pass, really in anger.

And, uh, and I tried to tell him, chef, and he, you know, he, we showed him chef and he loved it. And he said, you know, I, I, I think I'll eventually wind up buying this product, but first I gotta learn how to build this stuff myself. And, and, and, and he did.

And, and he, they did. You know? And, and it was like, we won't, I won't know exactly what I'm gonna need, you know, until I've gone through the pain, you know, that, that process of building this stuff myself and that, that's a, that's a heavy lift for most organizations.

But I love your term earlier, Patrick, the innovation tax. I, I think I, I, I really truly believe that, you know, that's what people are gonna have to do right now. They're gonna have to pay this innovation tax to become, to learn innovation.

And, and something, um, you know, that was said at, again, at the summit last week, which was you need to get on on the, you know, sort of on the train now, right? Because it's changing so fast. You know, I think there is a danger of missing some iterations of like learning like today what, you know, what, you know, retrieval augmentation looks like, which may look different in a, in a day from now, but certainly six months or a year.

Yeah. And so I think that theme of the innovation tax of, you know, join the dance now 'cause you gotta learn what you're gonna need to know, but, you know, as this sort of train is speeding up. Yeah.

Yep. You know, Patrick, can you describe a little more about that innovation tax? Like what, what, you know, being a, um, you know, run an engineering organization, you know yourself, like how do you get, how do you get people to adopt and think about a new paradigm without either derailing everything you're doing now, right?

Or letting this new paradigm just never or never happens, right? Just sort of gets kind of pulled back into the everyday work and you never really change. And I guess maybe this is sort of calling even back to your DevOps life, right?

Like it's, it's, I was gonna say He's done this a couple of times. Yeah. Universal problem.

But thinking with the lens on of sort of, you know, pulling generative AI into a already existing product organization. Like what, you know, how do you think about that innovation tax and, and you know, how do you pay it and how do you keep it to the minimum needed, right? Yeah.

When I, I mentioned the word innovation tax, it's, um, imagine you were a year ago, you know, very excited about this stuff, and you were building something with, you know, your search and you're embeddings and all the technical stuff. Yeah. Like three months later there's either a product or there's a new insight.

And so this means you're rebuilding the product again. So that is like a cost of effort that you went in. You put something out in production, so it's hard to change again.

And then you still have to do the innovation. So that's what I mean is it's, it's, if you want to be running, and, and I don't want to scare people with this, but be aware that if you put in a budget, uh, multiply it by one and a half because you know that you, you're gonna do things again like you don't know, because there, there's, there's rework all the time. So, uh, we could call this like continuous, you know, product or I dunno, continuous innovation, but that, that is the tax that I was referring to.

Um, and then obviously there's the tax of running two systems. So, you know, you're running this thing that you're trying to get into production that does one thing. And let's say it's a search engine and you have a cool thing and it's almost working, but at the same time, you're still working on your old search.

And then the example I I usually give is, your users then have to stop entering keywords that have to start asking questions. So both technologies, you've built them, you are at the fork, and then you still have to convert your using. So there's a lot of kind of, not per se friction, but things that you need to be aware, like, uh, while you're doing this, uh, on the go.

And, uh, it seems like it only going faster. Like you finally got something in production, there's a new way of thinking about this. Okay.

Square one. So, so that, that is kind of a, an intense moment right now, uh, of putting things out. And David, well, you know, one thing I was thinking about, you know, we, we've been working with Joseph Enox, right?

The over at E V T, he's been, uh, you know, really helping us all sort of understand. And I, and I think about an organization where, you know, in the past, you know, the way some organizations have created these guilds or whatever, where they're trying to get, like maybe this, the, uh, the, the enterprise architects in a guild, you know, this whole, you know, enterprise architects is not being DevOps or, or cloud native. And so they try to get the, so the fusion of like the enterprise architects are the right people to become aware of the emerging technologies.

And I think about, you know, every time we run into Joseph, he's telling us about a new paper, and I can't keep up, you know, and, and so if I'm in a large organization, I would wanna try to Kate like a guild where I've got a couple of people like Joseph who are just constantly looking at the research, following the papers, being able to decipher not every sort of paper needs, like to drop everything and rewrite, but to make sure, because it's moving so fast. And, you know, and it, it's like, um, you know, we have weekly calls with a, a few, about 10 people that are all involved. You know, we're, we're all trying to figure this out from our perspective.

You know, obviously you, Patrick, uh, you know, Joseph, a bunch of others. And, and that's how we're sort of keeping up. And I look forward to Joseph like saying, Hey, have you all read this paper?

And like, and usually when he tells you what paper, it's, so I, I would also encourage an organization to, you know, not only sort of embrace the prototyping, the thing like Patrick's, like try things out, get used to it, put a little extra budget in it, don't go wholesale anyway. So from a budget standpoint, it should just be r d, right? Um, so, so that's manageable.

Um, you know, don't start writing ELAs with vendors because they tell you that they can solve the whole world with their, their generative AI solution. Um, but also start creating those pockets of innovation, but then also create some sort of guild or something where you're just keeping everybody informed and informed in a way that the organization needs to understand it. Yeah.

So I think that's an important, uh, set of advice. Yeah. You know, it was interesting to talk to two, um, I guess one was a C T O.

There's a c i o one insurance company, one financial services. And they both made this comment that they felt like things weren't going fast enough this year. Right?

And part of the problem is they were doing this out of existing product teams, and they're trying to find a balance between, they want all the product teams moving, thinking like first principles, like how can I apply this technology to do things that I couldn't do before? Yeah. To, to, to actionize things that were either too hard or too expensive to do.

So we crossed 'em off the list. It's like, ah, I dunno how we're gonna, you know, too much variability from the top, you know, of the user intention or inputs or too much variability in the text in the, in the, you know, it's a lot of text in the world that they're working on. So they're trying to drive that.

But at the same time, they're realizing that, you know, they have this pyramid of run the business that, that keeps pulling those teams kind of back, right? So they're almost like token wins there, or easy things on the periphery. So they're both talking about, you know, like five, around 5% it felt like, um, of their total spend, they're just gonna carve off and put into a separate team.

Now, this is, these are all sort of, even though the one was a c I O, these are all very sort of outward facing initiatives, right? Whether it's supporting customers or creating new features to be able to drive, you know, new lines of business. And they're just saying they're also want to create a separate budgeted team that is gonna basically just say, go build products, right?

That product managers, you know, you're unencumbered by delivering on existing, you know, the business business line, we just need you to go build. So I think they're trying to find this balance between, you know, the importance of empowering everybody because that's where the serendipity is gonna come from. But also the importance of importance of trying to un unencumber people with the inertia.

'cause they're like, all of our competitors don't have that inertia of, you know, these billion dollar businesses. Like they're able to just, you know, uh, start from scratch, right? So, you know, this is a true like, innovator's dilemma of time.

Um, and it's interesting. I've just seen, there's a common pattern I've seen across multiple sort of industry industry trends thinking about this. I think it's, um, like any change happening or a transformation, right?

So you, you kind of have people who get excited, they're your champions. You have the option either to go inside your teams or bootstrap it outside because you feel there's too much friction on the inside, but if you bootstrap it outside, you have to find some way to bring it back inside, right? So that, that is always the challenge then.

And it's both on product and on engineering, right? Product needs to think about how can we leverage the new power engineers have to let go of saying, well, it's data, it's data science. We're not doing this.

You know, the news is it's integrations. You're very good at integrations. Please pick this up and you can make this happen.

Um, and then eventually, like you said already, that will move as a capability in a platform team that enables a lot of the others. So, so that will be a typical pattern, bootstrap inside, outside, but don't keep it outside. We've done that with data and it, it kept in a lot of ways being a silo.

And I, I think this is like, I, I wasn't, um, kind of confident enough to be in the regular AI because I felt like this is math and all the things that I need to understand and, you know, building a model and, you know, I need to go back to school for 10 years to kind of even get something on there. But from the engineering side, I read the docs, I can see it, it clicks. I can just start billing.

The cost is there, but it, it, it's a d click that you have to make, that you can do it. Much like, it was a little bit like, oh, I can do cloud. It's not that difficult.

There's an a p i, let's go for this. Right? And even the web, right?

Remember back in the day, it was like, if you could spell H T M L, you're a genius, right? And, and we tried to build a website that was such a foreign thing. We had to go there.

All these like razor fishmen were all these big consultancies existed to build websites because it just seemed like magic. And then when we all actually tried it, we're like, oh, it's not actually that hard. Right?

Yeah. I think I wonder if we're kind of can experience that same inflection point where people, you know, realize it's not that hard. Uh, the other thing too, I think is, uh, you know, I think about like, um, you know, the second time bringing up James Wicked, but he's got his new startup and you know, he's good friends with all of us and, and, and, and tech strong.

And, but he is got his new company called Dry Run Security, and he's been working on this idea for many years, right? He's, you know, started back with gauntlet and, and, but, and so when AI came out, I met him in, you know, um, in Chicago DevOps days, and he was trying, I gotta see this. He didn't rewrite his product.

His, you know, I don't know if his VCs told, would tell him, oh, you gotta do everything rewrite ai. He just added a little piece to it. Mm-hmm.

You know, so all the things that he's doing from this kind of context, security awareness stuff, he just added a little bit of a, you know, we could, I'd like to talk a little about, uh, retrieval, augmentation and rags, but he created a vector data store that, that had a lot of knowledge around the stuff that he, you know, that they do. And just added as a, as sort of a bolt on. And I think that's like another sort of advice to an enterprise.

You don't have to sort of sit back and start thinking, okay, what is next year's massively, you know, giant genive AI project we're gonna build mm-hmm. Give people the freedom to experiment. So if you've already got some remediation systems built or some, some, you know, generalized, um, automated governance type tooling, like, see if you can't just add, learn a little about these retrieval augmentations, eject the data stores, um, and, and see if you can't just sort of bolt on something that's useful, and if it's useful, add some more juice into it, you know?

And I think that, I think the, the another sort of trap people run into, we run into any new technologies, it's all in, you know, oh, we gotta go all in on the cloud, you know, that, that nothing drives me more crazy than we're a cloud first company. Like, before you think about anything else, you're not even like you. So, like, you're not, you're not gonna, you're gonna throw all other metrics, every variable, every business variable out, and you're gonna say, we're cloud first, and then we'll figure it out.

It's nonsense. You know? So I I have a fear that like, as people flip from the no, no, no, no, no.

To, to the now we're generative AI first, you know, you're gonna make these mistakes of, you know, trying to architecture design these massively scale solutions. Yeah. That's a funny, you know, it's funny, like, uh, even at, you know, PagerDuty rolling things out.

So we've got some pretty cool stuff coming down the pipe. People are like, wow, that's really cool. Right?

But one of the biggest, like, I need this tomorrow, right? Is, um, incident summarization, like, catch me up. You log in, you're like, what's been happening?

Right? And how many people have to stop? They gotta tell you something.

How many times they forget something or they tell you they leave something out accidentally and now you're kind of lost like that tax of just catching people up or that tax of letting stakeholders know every 20 minutes, let the business, you know, unit leader know what's going on so they can answer the customers. Like, like it's like, oh, just that. Right?

And it's like, that's the easiest thing for us to do, and it just got people jazzed up. Other things are like, well, that's really cool. I'll have to like examine that, see if it, it actually works.

And I don't know how to use that. But that instant summarization thing, you tell it to 'em and in 30 seconds if they're, you know, operations folks are like, yes, gimme that tomorrow. Right?

And so that's a good point, John, like, don't try to be, don't over rotate on trying to be profound. 'cause it's fun, but don't, right. Just fo just focus on being useful.

Right? Is that, I mean, is that, that's the way you'd, uh, describe it? Yeah, I think, uh, you know, the, the, you know, the productivity, you know, I mean, let's see it, there are, there are some big things in the future that this stuff is gonna, you know, lay out for us.

Yeah. But right now, I'd say view it as product. How do you, where do you find productivity performance gains?

Yeah, yeah. Quick. So We got a little about 10, 12 minutes left.

I was hoping to have more time for this, but you know, it goes, need to get the three of us together. But let's talk, let's talk about technology. Everyone wants to talk about technology, right?

Like, you know, I would look at the stack, stack and air quotes here. It's maybe kind of three levels here, right? I call like the prompt engineering level of things.

Then we call it like the middleware, orchestration, you know, level of things. And then the actual, like language models, right? So, you know, I don't wanna dive, I mean, we, you know, we have hours and hours and hours on each of each of these things, but maybe we can sort of talk about our thoughts around the high level of each of those levels and kind of what to, what to focus on, right?

And, you know, I'll kick it off with prompt engineering, right? I think this is something that, you know, often gets overlooked as how important it is and how far you can get in doing it. It's like, oh, I write Google searches, I'm handy, right?

Or, okay, I should write a longer Google search, right? Oh, okay. I should tell it to pretend it's a certain type of person, or I should give it an output format, or, oh, I should give it examples.

Or it's like the idea of zero shot or one shot or you know, few shot where you're, I'm giving examples, make it like this, right? And then there's maybe things like, oh, I can start to use, you know, tools tell it to use like a, you know, like a, a calculator tool or a, you know, a web search tool or something. Something like that.

Right? But, you know, I, uh, yeah, I just find that, you know, what's interesting to me is there's seems to be no depth, no bottom to how deep you can dive into prompt engineering and how important it is and how much you can get out of it, right. Uh, before, you know, attempting anything else at those lower levels.

In fact, I just saw one the other day where, um, the prompts were getting into like how to treat the tokens. I mean, it was almost like very prescriptive down into sort of the inner workings of, you know, how language model works all to get better performance out of that. Right?

Um, so yeah, I don't know. I mean, let's maybe start there. Do you have any advice for people on getting started with prompt engineering?

Um, you know, how critical it is or any crazy stories? Yeah, I'll, I'll take that one. Um, I guess it's, it's the one thing that gets dismissed.

Like you say, oh, I go on chat, G P T I ask it a simple question, it gives me something back. Right? Right.

But it is, so, um, the more you have a strategy on your bread, better prompting, it will drastically improve your quality of answers. Yeah. It, it, it is just, and again, it's not one prompt.

It's a sequence of prompts that that kind of builds up. And like first two, this strategy, then do that strategy. And we are, we're all kind of spoiled, maybe a little bit by, by thinking that, you know, chat G p t gives like this super magic thing, but if you start using and mixing it with your own data, you, you, you do need to put in a lot of work.

So getting the strategies better, there's another reason why this is, um, it's not important, but that has an impact. If you choose a different model, all your prompts are gonna react different. Hmm.

Correct. Uh, so if you were switching to a larger model, a smaller token, this is, uh, what I wanna point out is it's so brittle. Like if you do a change there, it could have a huge impact, but also a negative, huge impact.

So that's a very, that way. It's, it's quite important that you kind of keep track on how you're dealing with these prompts. Um, let's give the example of creating a quiz, right?

You know, you get, give me, this is the question, generate me four answers. And then you say, well, you know, generate me one answer that isn't a good one. And then you say, of course it tells you something, but it tells you something wrong.

So you can't use it in the quiz because everybody knows that's the wrong answer. Mm-hmm. So you kind of have to steer the model with some prompt engineers say, well, given this is the domain and given these are the answers, and this is the other thing, you know, make a better statement for the made up question.

So it isn't that obvious, so I'm just telling you, but you know, from a prompt engineering, this is quite a heavy thing to write in English and then to make it work on various models as well. So that's A very good, It's important in that way. That's an interesting kind of misconception that people think like, oh, these things are like, I don't know, you know, cloud instances, you know, it is a cloud, you know, a cloud machine, right?

Where it's like, ah, I, I spin up one e C two, I spin up one on Google compute. And there's little differences, but eh, it's a, you know, Linux dockerized thing, right? Like, you know, they're sort of fungible, right?

And I think that that your, your point that so much of this goes into the prompt engineering, that it's not portable, right? That you know that, that that one model thinks very differently than the other. So multi-model, right?

It's fine by design if you're doing different things, different things, but don't expect that that portability fungibility to be an easy thing. Um, yeah. So, uh, John, what's your thoughts on, uh, No, I think, yeah, I mean that's, uh, I mean, in a lot of ways, I think, uh, I can't remember who said this and I'm probably gonna mangle it, but it's like talking to, uh, an infant that learns infinitely, right?

I mean, when we have conversations, I mean, you in a sense, you feel like you are talking to a human because, you know, if I, if I meet somebody on a train or a plane ride, and like, what do you do for a living, right? Like, you, you're already, you're already starting the context of a prompt conversation. Like if we were just like, if, if every conversation when you're sitting next to somebody on a plane ride, what started with, uh, oh, I, I, you know, I, I go ahead and I do this C I C D stuff and I like to be, they'd be like, what the heck are you talking about?

Right? Yeah. So, but what you do is with humans, you literally say, what do you do for a living?

Well, I do this, I'm computer programming. And then they, they sort of like, oh, do you write code? Well, not all the time.

I more manage large infrastructure, you know, like, and, and now all of a sudden you've gotten that person 'cause what do they do? They make shoes or something like that, right? Right.

And you, you sort of work your way through. And I think the, um, the multi-model, and I I that a lot of the most interesting stuff too is these sort of having lots of different models. You talk about agents, right?

Agents that, I mean like somewhere in between like different models serving different purposes and then agents sort of using those models. But, but like imagine like, you know, again, if we go back to like the operationalization of like it and infrastructure and stuff like that, right? We might have models that are specifically defined that already know are in context aware of like, uh, governance and risk.

Yeah. And then we have other ones that are sort of related to incident. And then, um, and then we, you know, and then the other really important thing is this notion of what observability becomes in this new world.

You know, we still care about latency and performance, but we care more about this notion of evaluations. Yeah. The, you know, as it, it, am I running these sort of analysis against my model to say it's, it's 97%, you know, 97, 9% efficient in its evaluations.

And as I make more changes and as it grows, does the efficiency stay the same? Is efficiency for harmfulness or truthfulness, non hallucinations. So that's another whole, you know, to me that's one of the more interesting areas that I've been exploring is what does observability really mean, Right.

In the world of, you know, generative ai. Yeah. You you're talking about like, we spent 70 years trying to get non-determinism out of our systems.

Oh yeah, yeah, yeah. Totally. Now we're building it in.

So like, you know Right. That that's the, the frontier of how do you, which is I think what's holding back the agent rollout, right? Is like, how do you, how do you do the, you know, what's the, what's the process controls and the evaluation you're constantly put into to keep the Yeah, we, we should do another podcast because we did, one of the things we never got to is the sort of the dangers, you know, 'cause we're all, we were talking how great everything is and, and there is, uh, we should see if we could do a follow on our, because there, you know, I mean I think everybody on this call is pro on this, you know, like the answer's yes, go dive in.

But like, I don't wanna sound like we're not aware and we're not, We don't really understand that there, I mean the cars, the cars need brakes, right? You need brakes in the safety system. And that's the thing, um, in the last few minutes, like, you know, I I I, I wanna kind of talk about, um, you know, we use Lang chain as the representative sort of, of the middleware kind of world, right?

But it's this retrieval augmented generation, right? Is one of the patterns. And I think when people dive into the middleware side of this, like how do you chain together these different l l m calls with a little bit of open source code, suddenly can do all kinds of cool stuff, right?

And I, I I call it like the vagrant of, uh, of the AI world that when people first saw Vagrant, they just suddenly got the idea of the cloud. Even though it wasn't the cloud, it was VMs. They're like, Ooh, I get it.

You know, I just send you something, you type vagrant up, boom, there it is. You took Vagrant, destroy it's down. I type vagrant up again.

It comes up, it's like, ah, something makes sense. And people start to look at like the chat G P T, they're like, ah, okay, I get it. That's chat bot.

But when they start to get into sort of the middleware stuff that's emerging, they're like, oh wow. Like I get sort of, you know, how this is all coming down. So Patrick, I'll start with you.

You did a whole big presentation. Um, I've seen it go deep on the middleware world. It's a lot of options, a lot of things to think about.

Where should people be starting? Where should they be learning? What's kind of the, you know, 1 0 1 of, of chaining and middleware?

Um, I think the, the value of these middlewares or kind of orchestration frameworks is actually their documentation. This is the one that they have to keep up to date all the time. So for me, this was the, the most valuable place to learn about this stuff.

And then you go from, you know, the whole journey, the simple thing. I do a query to the L l m, I get an answer, then I, I have more some prompt engineering. I, I do more kind of a, you know, a bigger strategy to get like an answer back.

Then I mix it with my own data, um, and then kind of then I go, you know, maybe mix it with an agent or a tool that I already have. Uh, so that, that's kind of the journey that you can go through those tools. Uh, I have a GitHub library, which has a bunch of kind of lessons you can go through.

Um, and so I, I tried to explain it from the dev second, the ops perspective. So first one is the developer, and then like John said, the observability kind of making sure that, you know, how do you track all these metrics, uh, you know, cost, but also indeed like, you know, whether they're relevant or, or not. And then security, like, was this answer, did it leak?

P i i, uh, is it toxic? Is it leaking? Kind of like recommending your competitor on those as well.

So that's kinda, you know, a typical journey. You know, you start with the simple and then you keep, you think about the DevSecOps pipeline and then you think like, okay, what can we do here? What can we do there?

What aspect do we bring into that chain as well? Yeah. Alright.

Well are, we're, I'm getting the high sign to, uh, to wrap it up. John, do you con do you concur or did Patrick say anything that was completely, I disagree With everything that Matt has said for the last week. Now I agree with you, John.

Figure it out. I You agree with me. I wouldn't be here.

It wasn't for you two guys. So, uh, so alright, Well, hey, that's great stuff. com, uh, apparatus for, you know, helping put this, uh, this together.

And uh, John, I think you just agreed for us to do another one of these. So I think, yeah, I think there's a lot more to cover and this was fun. I don't, I hope the audience enjoyed it as much as I did, so, Yeah.

Good stuff. All right, well, hey, thanks everybody who, uh, stuck through this far and I hope you enjoy the rest of the, uh, the rest of the show. Thank you.