The Race for an Unfair Business Advantage with Damon Edwards at AIE 2024

Hello, everybody. Uh, welcome to the Executive and Business Track. Uh, my name is Damon Edwards.

Um, mark asked me to be the, uh, track chair for this conference, and I think you already heard from Mark and you heard from Alan. I think they did a great job of introducing what this, uh, conference is all about and what the unique sort of place in the world they're, uh, they're trying to, uh, to create. So, uh, perhaps I can give you a little more context about what we're trying to do with this executive and business track.

Um, and then I wanna, uh, talk about some trends that we've been, that we've been seeing, um, and then, uh, do a quick, uh, snapshot of the agenda for the day and get you on your way to, uh, to more content. So, um, you know, as I said, this is the executive and business track. The idea here is that, you know, there's a lot of content out, um, on the, uh, out there in the community that kind of is sort of in two polar ends, right?

One side is heavily technical. Here's how to use the LLM, here's how to build this rag, um, you know, pattern. Um, or it's all the way on the other side, right?

Which is, um, about, you know, large existential ideas of, you know, a GI and what's it gonna mean for the future of work and that sort of thing. And realize in the middle, there's a lot of people who are leaders who are trying to figure out, how do I actually apply this to improve my business and my, and my people, right? How do I help my people so I can therefore help my business?

Right? That's, that, that virtuous, uh, virtuous circle. So the point of this track is to start to collect a space where people can come together and start to talk about these topics and start to think about what is it like to be a leader, and what are the things that you need to worry about?

Some of 'em will go down the technical route, some will go down the management route. Uh, but overall we hope it's a great, uh, set of content that you'll, uh, that you'll enjoy. And obviously we welcome the feedback considering this is the, the inaugural year.

So, uh, you know, kind of onto some of the, uh, the trends we wanted to, uh, to talk about. Um, you know, in addition to my day job at PagerDuty, where I work a lot with generative AI and our operations copilots, and the, the products that, that we're building, uh, part of this, uh, I guess we'll call it like a loose, uh, community research collective called operationalizing ai. And what we do is we talk to a bunch of executives, and we talked to a bunch of folks that we know throughout the industry and try to find out what are they doing and compile these, um, these lessons learned, and then, you know, share them amongst each other.

So it's a lot of fun. And, uh, there's a few different, um, trends that we've been, you know, spotting as a, uh, as a group. And I wanted to kind of throw those out there and maybe get this conversation going, or get your thought process going for the, uh, day.

So we got about 15, 20 minutes here, uh, to, uh, to go over these. So the first trend I thought it was really quite fascinating to bring up, um, is that the user's user demand is unprecedented in enterprises, right? Uh, there's this interesting research that came out from, uh, Microsoft and LinkedIn research, I guess about a month ago now.

Um, it says, you know, 75% of all people inside an enterprise have already knowledge workers at is have already tried using AI at work, right? And about half of those, uh, started less than, uh, six months ago. So it's a pretty, pretty, uh, spectacular, um, uptake.

But something that was quite curious and actually quite scary to some people is that 78% of those people who responded, so 78% of that 75, so that's like what roughly 60% or so said that they're bringing their own AI to work right? Now, I know Microsoft, a, a, a, a vendor of AI tools will probably call that they call that BYO ai, bring your own ai, but for a lot of security folks, they call that shadow ai, right? So we're talking about now people are using their own AI tools in an uncontrolled way to do their, to do their work.

And, you know, there's been plenty of examples of history, whether it was, um, sort of the engineering side, it was the, the, uh, shadow IT starting in the cloud, or the people that bring your own device that really, you know, kind of got a headstart over what their companies could provide them. But there's an interesting component to this that I think is different. It's almost a bit of stealthiness or a bit of shame.

I'm not sure what word I wanna I want to use here, but of that, you know, 78% of 75, uh, the 78% of the 75% of people in, in, in an organization, 52% of those, um, are reluctant to admit that they're using AI tools for their job, right? So now there's a little bit of like hiddenness to this, right? And number one reason why they're, they're hiding that, right?

Is because they're worried that it's gonna make it look like they're replaceable, right? So, not only do we have this shadow AI thing that's, that's catching on fast, we have folks that don't even really want to talk about it, right? So there's a hiddenness to it that needs to be, uh, that need to be sorted out.

And of course, another major trend that we're seeing is, um, we'll call it GRC plus, right? Uh, as the governing governance risk, uh, compliance, there's a lot of issues, and it's important, and we've gotta sort this out, right? Um, here's a kind of, I did a rank ordering here of most conversations that, uh, I'll hear about, right?

Data and privacy, intellectual properties, compliance, both with existing and what's coming down the pipe bias, ethical concerns, even defining that, right? Transparency, you know, in the kind of control world, everything wants to be controlled, right? How do, how do we deal with this when we don't really know what's going on under the hood?

We have a hard time explaining it. Just the general non-deterministic nature of these, which then introduces these operational risks, right? We now have new operational risks because we're injecting non, um, uh, non-deterministic systems into our formerly, you know, very deterministic systems, right?

And then it's not just for us, but there's these transitive risks from vendors, right? Like, now we're using these vendors. What are their policies?

What are they doing with our data? What are they doing with the service that we're using from, uh, from them? So all these things, you know, they're, they're very important and we have to figure it out.

But at the same, same time, we're seeing in some organizations, it's becoming basically debilitating, right? There's an analysis paralysis. They, they're, they're frozen.

Um, and it's really having a hard time getting things going because of this, you know, all of this, um, uncertainty. One of the most interesting, um, things we've seen is the companies that are kind of getting past this. Well, you know, I, I don't know if you recall from the, you know, the, kinda the last 20 years of software development, this notion of shift left, right?

Which is like, how can we, uh, take a lot of the operational concerns, testings, concerns, security concerns, and bring it forward in the development process? Well, they're doing that with GRC, right? They're saying upfront, let's figure out the privacy issues.

Let's upfront, let's figure out the data and the IP and all the other, uh, um, you know, issues that we're gonna have around this. Try to preload that work, um, on actually the product and development side, instead of waiting down the road for, you know, legal to come in and, and, uh, um, and what is invariably been quite an issue. So, interesting thing we've seen there, huge emphasis on it.

Some folks are figuring out out ways to get past it. Other folks is really becoming a boat anchor that's, that's, uh, uh, that's, uh, blocking what they wanna do. So another trend, uh, we've been seeing is, uh, you know, I, there's this data from, um, Andreessen Horowitz, um, and we've been seeing the same thing, right?

That, um, you know, on average people are doubling, tripling, uh, the spend that they spent last year on LLM and LM related, or, you know, generat eye related, uh, technology and services. And, um, you know, that's, that's not, uh, uh, to be, um, unexpected considering all the excitement in this area. But what's very interesting about it is people are sprinting so fast.

There's really, you know, like according to this data, which I think is quite true, like 60% don't really have an idea of where their ROI is gonna come from. Either they're not looking at it at all, or they're just know that it's gonna be good, but they're taking it on faith and not sure where it's gonna, it's gonna come from, right? And you know why that's, that's, uh, gonna be quite, quite an issue is, you know, where is that money coming from, right?

And I think it's, see a couple of surveys like this that show that a lot of that, the money for these AI projects, it's being carved out from existing budgets, right? So in the recent Horowitz survey, they showed that only 20% or 19% here of this money is net new money. It's all being reallocated from other, uh, from other budget sources, which is fine when you're getting started.

It's fine when you want to kind of, you know, uh, uh, you know, borrow from one one to pay the other, right? Um, but it's gonna be an issue as things grow, right? You're not gonna be able to cut and carve from these budgets forever.

We need to have a clear ROI discussion on where's this gonna come from, where's that investment gonna come from? And then, you know, no investment, uh, really can be made unless there's some, you know, ROI theory, uh, behind it. So it feels like the ROI, the ROI, uh, analysis and theory has to catch up to the actual spend, uh, spend issue right now, right?

So, what do we see people doing, right? What is the short term thing that I think has been happening across mo most companies, um, you see them going back and looking for their, their, uh, their previous, uh, things that they said no to, right? Because they were either, you know, the inputs in the user intentions was too variable to unpredictable.

So it's just too hard and too expensive to figure out how to do. It wasn't a strong enough case for it. Or the underlying data they're working on had too much variability, lack of structure, unpredictability.

So therefore, you know, it would've been too expensive or too, or too costly to do it. Or they just knew that, hey, it would be great if we could automate this, or great if we could, you know, have the machines do X or Y, but it's just too hard to create that software, and it's just gonna be too complicated. It's too expensive, and based on other priorities, we just can't spend this money or time, so we gotta cut it off the, uh, off the list, right?

So, you know, and now it's like, hey, you know, the, the question we see folks is going through those old lists, going through the old ideas, the ad to say no to, and saying, Hey, can we use generative AI to solve these new problems at a fraction of the cost, um, and a fraction of the time that it did before, right? So it's basically rethinking, uh, recent history to say, Hey, these decisions we made, are there decisions that we made? Um, because it's too, it's too expensive, we just couldn't do it.

It's, it would take too long, and now we can actually go and do it. So we see a lot of short-term, uh, you know, emphasis on revisiting those, those old, uh, those, those old decisions. Now, you know, what do we see people looking at when it comes to long-term, right?

Uh, long-term, really, especially in the enterprise, it really comes around, uh, the transformative labor impact, right? I think there's a feeling that a lot of what we see on the kind of customer facing side, um, is, uh, there's such quick parity, right? When somebody comes out with something and somebody else can, can do it reasonably, reasonably.

Similarly, we haven't seen a lot of really, I think, groundbreaking ideas yet. I'm sure there will be, um, that are, you know, in, in how to revitalize or, or, or rethink our customer facing, uh, products. So we see a lot of folks think, uh, looking internally first, right?

It's also safer, right? All those GRC um, issues that they, that we listed off before, a lot of these are, you know, less, uh, less of a concern or can be more, uh, controlled if further looking inward. So pretty simple formula, right?

They're looking, where can we avoid people, right? How can we assign, you know, protect our people from work that's not the highest value work or work that you know, that, uh, we can get these, uh, machines to do for us? Use different people, right?

Can we, you know, help guide people through work that previously they would have to escalate to a subject matter expert or they're not qualified to do? Or can we have a different cost level of, of people doing this work, again, to take our best people and put 'em on, on things that'll move the business, uh, forward. And, uh, you know, and then just making our people faster.

This is the whole copilot, uh, you know, mechanism, right? How can we just turbocharge the folks that we, uh, that we, uh, that we have? And so, you know, what's the sort of timeframe on this, right?

I think today we see people looking at how can we replace tasks, right? So look at anyone's job. What are the tasks that you could help with, right?

So you see a lot of the coding co-pilots. Um, you know, what I work on at PagerDuty operations, co-pilots. How can we be the sidekick that helps do the tasks that is gonna make those people, uh, better?

That's something that, you know, I think is possible now and, uh, is actually quite, uh, quite useful. What's going on tomorrow? What are we trying to get to?

What's on the horizon is being able to replace entire job functions, right? So just say, Hey, if you had a, you know, a senior engineer or a operation, some of the operations side of the house, the business finance, can we say, what are the parts of their, what are their functions in their, in their job that we can either fully automate or paralyze, right? Have multiple, um, you know, uh, iterations of that going at, um, at once.

So really looking to take the next step, which is how to replace job functions. And then down the road, I think, which is something that's purely theoretical at this point, unless we're talking about single function jobs, but being able to replace entire, uh, you know, entire jobs, right? So instead of having one person do a job, you have one person with, you know, a hundred agents also doing that, doing that, uh, doing that same, uh, that same job.

And you know why? 'cause like, we're talk, we're seeing results today, especially around like copilots 30, you know, 40%, uh, and more, um, eff efficiency gains by replacing these tasks. So, you know, tomorrow, can we get 10 x of that, right?

If we're able to replace entire job functions, um, but then where does it go from there, right? We can we 10 x again from, uh, from there if we're able to replace and kind of replicate entire, entire jobs. Now, you know, I think what we're gonna need is some serious advances in, you know, agent technology, um, to get these last two.

But, uh, I think that's where everyone's thinking. At least we've seen 'em thinking on the horizon. So task today, jobs someday down the, uh, down the road.

And you know what this brings? Another trend we're we're seeing is people are realizing that the urgency is a little bit different this time, right? Um, you know, in previous investment cycles into new technologies, you know, there's things like the web, right?

It was cost additive. You had to build these, get these new servers, people with new skills, you'd build some websites. It was all a, a bet on a future strategic, uh, direction, right?

Mobile as well, right? It was cost additive, new infrastructure, new technologies, new servers. Um, you know, and the strategic benefit was kind of somewhere down the, uh, down the road.

Very interesting about this AI is that we're actually talking about it's cost deflationary, right? That we can, can we start replacing tasks? We start making people more efficient, uh, right away and get this immediate, immediate strategic benefit, right?

So, um, you know, margins are a very important thing in any business. If you can do anything that immediately starts attacking those margins, growing the margins, keeping our costs, our costs lower, that's an immediate strategic benefit. 'cause the company that has the, um, you know, that advantage when it comes to margin is gonna have an advantage overall, um, with their, with their business.

And we also see this trend now where when one person in the sector figure, or we're expecting when one person in the sector figures it out, everybody else has that, has to jump and follow too. Because again, you've got that strategic advantage of, of, uh, you know, of better, better margins. And, you know, always leads us back to like, Hey, someone's gonna lead it for your industry or your sector.

You know, why not? Why not your company? Right?

And a great example of this, um, this kind of happened back in February just a few months ago, right? Uh, which is ancient history in the AI world, right? But pretty recently in terms of the calendar.

So Klarna, you know, the buy now pay later giant, uh, they announced that, uh, they're going to, they created these ai, um, assistance, um, they created it with the help of OpenAI, but is their own engineering. And, uh, they're now to deflect two thirds of all customer service chats in the first month of turning it on, right? So that's gonna save 700 agents for carna that goes right to their bottom line, gives them an advantage.

And the NPS score of those interactions actually has gone up, right? So pretty big, uh, deal, but was really fascinating is at that same time, same day that this was announced, klarner, this press release, uh, um, tell their performance, which is one of the biggest, uh, I think maybe five, 600,000 people, uh, call center operators in the world, um, their stock dropped 20%, right? And on this, on this news, right?

And, uh, it got so bad that the tele performance CEO had to go on CNBC and announce that, you know, this is, they're doing AI too, and you know, the world's still gonna need them, which is never a, never a good luck when your executives have to do something like that. So, very interesting how, um, you know, such a quick impact has had when, as we said, when one person in industry jumps, everybody else is gonna have to jump, uh, jump with them. Uh, another trend, right?

We've seen is that this, uh, you know, we have no boat, right? Um, you know, this, uh, came out in May of 2023. It leaked out.

There was this Google memo that said, Hey, uh, you know, they're arguing that, um, all them and all the closed models model vendors are in unwinnable arms, race against scale, distributed, open source, uh, innovation, right? And the idea is basically that this technology, once it's open is just, is, is amazingly easy to use and get and get going to where that anybody can do it. You'll need to have just the, the greatest resources to, uh, to do it.

And, you know, I've seen some other interesting examples of this, right? Uh, Joseph Phoenix actually is gonna be speaking later today in our, our track. He, um, you know, took that paper and he really liked the perplexity ai, their search engine that you can ask in a question.

It does research, gives you results. He said, can I do that right? And, um, you know, he's not a professional developer.

He is handing some Python, but using examples from GitHubs and Lang Chain, you know, the G-P-T-A-P-I, he basically landed pretty much on the green next to what he got with his results from, um, you know, from perplexity with, uh, you know, a whole lot more financing and a, in a, in a large, much larger team, right? So it's one of these things where super easy to get started. Um, and I think people are realizing this, and, you know, even though this we have no moat idea has been out there in the past, it's really starting to jump up, uh, Devin, right?

The, uh, the, you know, the first software, AI software engineer, huge flash with the, with the demo, huge valuation. Um, well, you know, within days there were, you know, thousands of people on Discord saying, how can we do this ourselves? Within weeks, there's, people have already created, uh, clones, alright?

Other ideas of AI coders that they think are outs are outstripping or doing can, can, uh, beat, you know, Devon on these, um, different leaderboards, right? So again, it's one of these ideas that, you know, that, that the, the, even the best ideas, the moat in this area is so small and so fast that people can, can catch up. I think it's really surpri catching people by, uh, by surprise.

Uh, another trend we've seen is interesting is that people who are, uh, you know, the first thought was, well, hey, let's just have the LM do everything, right? Let's just replace all of our business logic. Just try to, you know, uh, use prompt engineering, fine tuning, make the LM you know, do do everything right?

And very mixed results, easy, kind of fast start, but didn't do too well. Um, kind of in the long run, we're seeing people rethinking things, right? Saying, Hey, how do we use the LLMs to handle the user intent and all that fuzziness around there, and then handle the looking for, you know, working on data and all the fuzziness down there.

And in the middle, we'll kind of stick to our traditional app logic, app rules, you know, traditional ways of building, uh, software. So seen a lot of, uh, on the left there, a lot of early initiatives or early, uh, pilots started there, um, but quickly kind of rethought things. And so we got a little more ambitious, more of a hybrid approach of architecture is where we've, uh, we've gotta be, um, retrieval augmented generation and won't go into it.

I'm sure 10 people will on this, uh, uh, in this conference. But, you know, it is a thing. Everybody starts with the ability to put documents, you know, embed them, put 'em in a vector database, have the LLMs, you know, when you ask it a question, search it for its knowledge, bring it back, recycle it through the LLM, provide you with a, an answer that's grounded in actual, you know, kind of facts, right?

So this is the sort of one of the go-to patterns that we see, you know, in, in every enterprise trying. The one that's, that I think the more forward or progressive ones are trying as well, is the addition of adding an automation manager, right? How to use, you know, a building, um, as code, a capability manager that knows how to pass these various, uh, various chain that says, Hey, take the prompt, decipher the intent, match an automated capability.

We have defined sugar, that capability, interpret the results, decide if it matches what we wanted to do, and kind of continue to loop from, uh, there. So, uh, less in the open source realm, I think around this, but, um, uh, we're seeing a lot of people inside organizations, this is kind of the next thing they're focusing on building is marrying, rag with, uh, with, uh, with automation. And like I mentioned before, it's this theme that like, you know, uh, funny people that wow, it was so easy to get started, um, but then when it came to actually, uh, you know, perfecting it, things were really, really hard, right?

This is the CEO of, uh, Cloudflare mentioning that. And of course he was talking about in the context of VCs, but I think people are finding it in terms of just, uh, um, you know, their business in general, right? And a lot of that is because kind of what, what are these things, right?

They're, you know, the LLMs are non-deterministic by design, right? It's, it's not a, um, you're gonna get randomness there at any point. It's a lot of fiddling, right?

You're gonna be working with, you know, how do I do the queries? How do I split the text? How do I, or the chunking, right?

How do I, what kind of embedding am I gonna use? What's the metadata? And putting in the vector database?

So it's a lot of fiddling, right? And then you kinda get to a point where you're like, okay, that works. Now nobody, nobody touch it, right?

And of course, these kinds of systems have big challenges, right? They're, you know, kind of change anything, change everything. Um, a lot of the tools they're supporting this are old ML technology, right?

It's very kind of desktop or work group oriented. Um, uh, not made for dynamic kind of web scale, um, solutions. Um, you know, you have a lot of uncontrolled data dependencies problems, right?

Change something here, it's gonna happen over there, or you don't know somebody change something over there. What's gonna happen over there? Um, and then of course, we have to monitor these non-deterministic systems and, you know, which gets to another big, a big aha people are going through is what does quality assurance look right, looks look like, right?

And we spent 70 years of trying to root out non-determinism, right? From software, right? Back in the old days of what was a bug, it was actually literally a bug that caught into a vacuum tomb, right?

That, you know, screwed things up. So we're trying to, um, you know, get rid of that, right? But now we gotta rethink everything from a non-deterministic, ByDesign, um, nature, right?

And so for testing, when things are very deterministic, it was easy, right? We're easy, straightforward. So we said, Hey, developers, okay, we built something, did it work?

Did it pass our test? Yes or no? Right?

Then we go into production, we have a different kind of testing we call a monitoring. Is it working? Yes or no, right?

And unexpected things might happen, or tests we didn't make, we didn't know for, but we can keep layering on these tests. And we would have this very deterministic as a worker, does it not work? View of the world, but in this new world, what does it mean?

When's on these non-deterministic, well, in a development environment, we're gonna get things kind of good enough, right? But we can never really a hundred percent say, sure, this is gonna be a yes or no type, type answer. And then production same ways.

It's not gonna be a pure yes or no, right? It's gonna look something more like process con, process control plus multifaceted evaluation, right? Being able to keep these things within control, evaluate answers, understand where they fall on the, um, on the range of, uh, uh, of acceptable what's not acceptable.

And, uh, being able to, you know, kind of keep it under control more than say yes or no. Things are good. So I think QA is totally being rethought around these things, and a lot of people are finding that they're gonna spend more tokens trying to control these services than they are tokens actually producing the first result in the first place, right?

I guess how dramatically different things are. And, uh, same way with, you know, we see people saying, we gotta fundamentally, for the first time in a long time, reinvest in our user experience, right? It used to be a lot of tasks, a lot of data views, right?

A dashboard view of things. But now it's conversations, right? What does that mean?

What is it like to work with another person? They don't talk to you through a dashboard. They talk to you through a conversation, or they show you pictures or, you know, it's very, I think, an emerging area.

And we're seeing, um, I think a new, uh, invigoration investment in, um, uh, user experience that's fundamentally different than was done done before. And last one here, just, you know, fundamentally, um, you know, uh, non-determinism just bothers a lot of people. We're still seeing this and people hearing this, that they're still surprised that which people in our organization are gonna not like this, and which people are gonna get excited by it, right?

And, um, you know, I have a partial theory. I think, you know, part of this comes down to there is job loss and change, and, you know, who am I and what's my value and what is intelligence? There are some major issues like that, but also coming from the, the DevOps world, you know, I, I, I saw this a lot where, you know, in the, um, uh, dev side of the house, things are very, you know, non-determinism is a bug, right?

It's that you e either it, the software builds or it doesn't either deploys or it doesn't either gets the correct answer in production or it doesn't, right? That's sort of a very developer deterministic view of the world. People come from more of an operations background, say the world's a lot more messy, right?

That, um, you know, it's a complex stochastic world. We can never really fully understand these large complex systems we've built how they work, what's gonna, what impact the change is gonna make. And they learn a lot more to deal with that, with that, uh, non-determinism.

So we've kind of found that it's an interesting thing that people who have a more deterministic software oriented bent, um, tend to have one view, and people who have more of an operations we're fiddling, as always been the rule. Um, and we've been trying to, you know, root to, uh, non-determinism out, um, you know, tend to have a different view of all this. So it's very interesting.

I think that we see a lot of leaders trying to figure out the psychology of this and how to approach their troops and how to, you know, really understand kind of where who's gonna be a supporter and who's gonna be a, uh, a much more cautious about it. Okay, well, so that's enough from, uh, from me, hopefully some of those ideas or things that might have kind of got you thinking. Uh, if you wanna talk more about those, uh, I could love to talk to, uh, anybody, uh, anybody about it, some really interesting, um, things we're learning.

But I wanna talk to you about the rest of the day. Why are you here, right? Uh, I think we got a great track, um, lined up.

We got a full day of, uh, of sessions. Um, you know, starting off with, you know, cybersecurity. It's a huge issue.

Um, you know, people are using these, uh, new technologies for bad things, right? A imagine when you can try to do it on the exploit side now, but also for good, right? We're gonna have to build monsters to fight, uh, to fight monsters, right?

Um, and, uh, a lot around, you know, Strat AI strategy, um, you know, how people are doing these early projects. There's a keynote, uh, from Ruben C's doing fascinating work in large enterprises. Uh, I think a lot of great things to, uh, to learn there.

Um, I'm doing a talk, I thought it was a interview, uh, somebody who's, uh, really spectacular, Jody Mulkey, um, a long time, uh, CTO, um, and sort of, you know, their, uh, his kind of field report of a leading organization who's, you know, jumping full, full force into, uh, using generative AI for their, uh, for their business. Um, and, uh, yeah. And so we're gonna be going on.

And then at the end, uh, I mentioned Joseph Enix, right? Another, uh, uh, kind of friend of the organization here who, um, is another, uh, great talk about his experience from working inside large enterprises, early movers with, uh, uh, with generative ai. So I've talked enough.

That's it for me. Um, I hope you all enjoy the day and I'll see you around in the, uh, in the chat. And, um, enjoy.

Thank you.