AI Experimentation for Business Value with LaunchDarkly’s Claire Vo

Hello, and welcome to the latest edition of the Techstrong AI video series. I'm your host, Mike Azar. Today we're talking with Claire Vo, who's chief product and technology officer for LaunchDarkly.

And we're trying to find the balance between, well, all these things that we're excited about doing with AI and all the risks that we all know far too. Well, Claire, welcome to show. Oh, thank you so much for having me.

I'm very excited to talk about this. I think we went from this period of irrational exuberance with the ai, and we were all these awesome things that we could do. And then we discovered along the way that, well, it's not, you know, omnipotent, right?

It's probabilistic, it is giving us the best guess and recommendation. And now we're all trying to figure out how to insert that into a lot of processes, some of which are, shall we say, highly deterministic, in that they gotta be done the same way, the same fashion a hundred percent of the time. And I think we're struggling with this, right?

Because it creates a level of risk inside of AI that we're trying to find a balance to. And do you have any insights as to how folks might wanna approach that? Yes.

So I'm gonna, I'm gonna say one thing that may surprise, may, it'll surprise you, it might not surprise people that know me, is I am still in the phase of irrational exuberance despite all the risks, despite, you know, as I say, non-deterministic system, non determines de despite the fact that quality is challenging and risk is there and it's not perfect, and it doesn't know anything, and things still hallucinate, I am still in this mode where it is just very obvious to me. This is a state change transformational technology moment. And, uh, I actually think there is under appreciation for how transformative it can be.

And I think risk and quality are being used as shields to actually embrace the full power of things or really understand how you can get to high quality outputs using these non-deterministic systems. So by inviting me to this conversation, you have probably invited the most bullish AI builder out there on the topic. So I'm gonna anchor everything I say and that I'm very far on that side of things.

That being said, it is really challenging to build with these new technologies, especially in I think, three frames. It's challenging from a technology perspective to build software that has classically been built on deterministic algorithms with non-deterministic large language models. That state shift in how to make your core technology work is not a straightforward proposition, especially when you have end users on the other end.

So it's problem one. I think problem two, it is a very challenging, and it is a real, um, learning and development effort to upskill software engineers and their, you know, partners across an organization, whether product managers, DevOps, SRE, security, et cetera, on these technologies and how best to apply them with the right guardrails, best practices, risk mitigation pieces in place. So you have, you know, everything from, you know, the person that tinkers with stuff on the side and thinks they can just put anything into production to people that say, this is a fake technology, it's a bubble, and I don't need to learn it.

So there is like a real cultural and skills upleveling that needs to be, needs to happen at, uh, at a talent level. And then I think operationally, entire companies are trying to figure out, how do I operate with this technology as part of our stack? How do I, um, ensure quality?

How do I ensure the right human in the loop practices? All those sorts of things. So I certainly think there's challenges across the board.

What I think is good though, is these challenges are not new. We've always had to grapple with how to integrate new technologies into, into an existing software stack. And, uh, what I like about LaunchDarkly is what we're finding is some of the best practices for, you can't even call it traditional software develop.

Modern software development practices are just as applicable when building things with, with LLMs and, and with models, which is, you know, you wanna make sure you have the right runtime controls, you wanna make sure that you can roll things out and make sure that you're experimenting both to, um, protect downside as well as maximize upside. And so I think the mental models are actually quite applicable in this new world. We've just got to give teams, engineers and companies apps to get their safely mm-Hmm.

To your point, I think people look at the large language models developed by OpenAI, Microsoft, and whoever else is doing that, and they don't realize what went into that. There are, you know, millions and millions of dollars invested in building out that capability, but it's a jack of all trades kind of capability. And, um, the average enterprise isn't gonna go build that, but they can customize it and they can extend it.

But I think when they get into this conversation, they kinda realize that they don't really have the data organized in a way that allows them to easily extend those things. So how much of our current situation is really tied to, uh, data management, data engineering challenges before we can kinda leverage the LLMs? You know, this, this is a good question.

Again, I would be curious in most applications, in most organizations, if that is a true barrier to the first couple wins of in integrating these technologies. I mean, the reality is these commercial models have been developed with quite a robust, um, corpus of data. And it is, they are fairly good at, um, through context and increased context length, actually operating pretty well without fine tuning for many use cases.

I'm not saying every single use case, but I do think people are underestimating the power of these models, which if you haven't been paying attention, are being upgraded every week, every month, where context windows are expanding out to the size of a book where I think folks are really, um, saying, well, we gotta put our own data in here without actually experimenting how far the generalist models can get you. And this is where I think, um, the operational practice of developing AI products is really a new world for folks. Product managers don't know how to define products that rely on non-deterministic models and don't know, don't have a, a guidebook on how to experiment their way to determining which model is gonna create the right customer outcome.

That could be a fine tuned model. And then you need better data, infrastructure data, better data, cataloging, better data structure. But it could be you just need to mess with your prompt a little bit.

It could be model A is much better for your use case versus model B, but again, we're not giving teams that are tasked with building these products, the right guidance on how to discover their way there. And so they have these big questions like, can I even start without the right data infrastructure? Is it even appropriate to use a commercial model?

Which commercial model should I use? When would be a smart time to host or an open source model, or use a hosted open source model? So I think the questions are there, I just am approaching it from a different angle, which is let's talk about the customer experience, what we want our end user to have on the out, out outset, and then how do we work our way back into the technology set that delivers there as opposed to starting technology first and working our way into a user experience.

I'm not even sure you're describing a technical issue as much as it is a cultural issue. I think almost every day now, I say to somebody, you know, you could do that yourself using chat GBT. Yeah.

I mean, I'll give you a really specific, quite, I mean, truly very timely example, which is here at LaunchDarkly for new, um, hires, we run an architecture 1 0 1 presentation. So every month engineers sit in a room and they spend an hour with new hires, they walk through a set of slides about our architecture for our flag delivery network and our data pipeline, et cetera. We're a, we're a, you know, a developer tools product is important that everybody in the organization knows how our architecture works.

And I just saw a list of kudos come into one of our Slack channels that said, this is the most valuable meeting I've been in. I got so much more information, I learned so much. I wish I had had this information earlier.

I took the transcript of the last three recordings of that meeting. I took the presentation from that meeting and a couple other, um, architecture docs. I uploaded it to our corporate Chad, GPT made an architecture 1 0 1 GPT, and now anybody can ask any, any question they have about our architecture to that GPT without waiting for the meeting.

And I will say I did get a, a validation from one of our best engineers that it is actually quite, and he says quote, quite impressive. So I, I do think, and it took me 90 seconds to make. And so it is, it's a little bit of culture.

Do you have the culture to embrace automation, to embrace transparency, to be experimental? I think that's very critical. The other thing is muscle memory.

People are not yet used to these tools as a go-to tool. You know, you're used to opening your Google Doc or your Microsoft doc, you're used to going into Slack. That's muscle memory for you.

These model back tools, these chat tools, these a AI tools are not yet muscle memory for folks, even if they can get the job done. Mm-Hmm. Yeah.

And the subject matter experts that I talked to are getting annoyed. They're kind of like, Hey, I've been answering these same questions for years now, and there is a way for you to get the answer yourself. So, you know, give all those subject matter experts that need to go do some real work, you know?

Yeah, right. Exactly. All right.

But we would like to get to the next level. And you hear about the rise of, uh, a agentic ai, which to me is, uh, rag on steroids, right? I I have a, um, a corpus of data that I've used to train a, uh, smaller LLM or a domain specific lm Yeah.

On a task. And I'm leveraging its reasoning engines to perform a series of tasks. Yeah.

In theory. Um, how smart are those AI agents getting and how good are the reasoning engines? You know, one of the things that I think we need to do is demystify what agentic systems really are.

And the best ones that I've worked with work the same way a well organized human would. So this is what's really kind of fun about working with LLMs, especially as somebody who's candidly has made their career in people management and people leadership is they operate very similarly. And like you, you know, you say it's rag on steroids, I say it's just a very organized project manager that can do a series of very tactical tasks.

All these agent systems, generally the ones that I've seen that are most effective, follow the same framework. They get asked a question and they, the first thing they do is say, if I'm asked this question, build a plan to answer the question. Step one, step two, step three, step four, step five.

Once I build a plan to answer the question, build me steps, a plan to execute each of those steps, and then I can pull from a list of tools, whether it's two tools or two dozen tools, to get that job done. So it's, you know, it doesn't have to be a mystery. It is a, you know, dynamically planned system, which is quite fascinating.

It is intelligently selecting from tools using probability, but it is not a black box. You know, some parts of it are black boxes, but the ENT system itself is not necessarily a black box. So what I have found is that, um, ENT systems, the ones that I actually rely on on a day-to-day basis, are at about a solid intern level of independence.

And so I'll give a very specific example of an agent system that I use relatively frequently, which is I use one of the most, um, more, uh, talked about, uh, coding solutions, Devon, um, which said a lot of debate about whether it's real or fake. It's actually quite good at PR review. So I have a, a loop, I have a little side project that I do.

Um, every pr I have Devin, go look at the code, make sure it can run, make sure it's well documented, document the PR comments, like something that you'd probably trust a, a junior engineer or an intern to do. And it does it quite well. But in the same turn, those kinds of systems sometimes get stuck the same way.

You know, somebody who's less independent would get stuck, Hey, I'm having a trouble setting up this part of my environment. Hey, I'm not quite a understanding how this loop works. And so, but that being said, a $50 a month intern that I can rely on at any moment of the day that could do a solid set of work for me, that takes that task off my plate so I can focus on something higher level, what tremendous ROI, for me, it's just, it's huge for me.

So I do think, and these systems are so new, everybody's like, oh, they're terrible. They don't, they don't get a job done. They're brand new.

Think about where we were a year ago or a year before that. Like, I try, I'm trying to operate as if I'm planning for what I think the world looks like in a year, three years and five years, as opposed to what the world looks like now. And if I have confidence that these technologies are getting better, I'm gonna operate presuming they're getting better, and figure out how to use the tools in our workflow As we have more of those agents, which is at this point is a given, I think mm-Hmm.

How will we orchestrate them? How will the agents, uh, know how to interact with each other? 'cause I want, you know, my AI intern to talk to your AI intern to perhaps negotiate something or come to some resolution and then, you know, let me know how it all turned out.

Yeah. You know, what I think is really interesting is, um, as much as AI products allow non-technical folks to build applications, I still think that to build something meaningful right now, um, you need to be able to think like a software engineer. And so I do think these systems, these agen systems will be built on fundamental software engineering concepts that already exist.

These are really just going to be API to API calls with non determinate, the non-deterministic inputs and non-deterministic outputs on both ends. And then what we're probably likely to have is given that these agents will all wanna operate API level, but that the outputs or structures of the data are going to be quite different is I am presuming we're gonna have some smart automation in the middle that massages API endpoints into whatever the next agent seemingly expects. And there's like a little loop of, as opposed to a deterministic schema, non-deterministic schema that can actually talk to whoever you're gonna talk to.

I don't know, maybe I just came up with a great dev tools, uh, idea and I should go build that. But I do think it's really gonna be like API to API systems. At the end of the day, those APIs may just take in natural language to natural language.

That can certainly be the case. And then from an orchestration perspective, I think, again, let's use the same mental models we're used to in modern software delivery versus AI software, which is you're gonna think about APIs and you're gonna think about integrations. So every agent is gonna have a supported set of integrations with other agents, whether you build those yourself or they're supported by like sort of a cloud hosted agent provider.

And so I think we'll see the same kinds of concepts come into play. What I think that means though is that the proliferation of one-off agents will be limited for the next, I don't know, call it 24 months until we all figure out how we're all gonna talk to each other. Until then, you're gonna have kind of single agents working within a system, um, a probably very little vendor to vendor handoff in the short term.

Although long term I think we'll figure it out. But I think we're already seeing scenarios where there are agents that are creating code and then there's another agent that's reviewing the code. Yeah.

So that there are kind of working in some sort of hand in glove fashion. I just wonder at some point when there's a conflict, will we have like AI agent referees that are, you know, Again, I mean, what I think is kind of fun about all of this is, um, and maybe why I am so optimistic about it is, look, these models are hu chain trained on human be human language, which ultimately comes out of human behavior. And, and so what I anticipate is human skills shall apply negotiation management intervention.

I think the hardest thing that will, it'll be hardest to recreate is spontaneous creativity. Um, and sort of like true, true novel insight I think is gonna be challenging. Um, but yes, we're absolutely going to have management issues with agents.

And I've actually spoken about this a lot. I think man, people leaders are gonna have to start thinking about a world in which they manage people and they manage agents, and that is their team and their job is actually to figure out how technology and humans work together and figure out the right solutions as opposed to only thinking about the, the people side of things. So to that end, well, we send people to school for management training.

Will, will that whole thing have to evolve where we learn how to manage people and agents? A hundred, a hundred percent. I just cannot say how important this shift is going to be for man and managers.

And you know, what I also find really interesting, uh, you know, CCPO at LaunchDarkly, I think many executives think it's coming for their team and it's not coming for them. And I really would urge folks, especially in senior leadership organizations, to think about what is the organization of the future gonna look like? And I'm operating that it's gonna come faster than people expect.

Other people are off, you know, saying it's gonna come slower than people expect. I'm, I'm just operating, it's, it's gonna come faster. I'm thinking about what does that mean for my headcount planning?

What does that mean for my shift between software spend and people spend? What do I need to coach managers up to do? Who's going to be a great manager in that organization?

And then how do I use AI to get leverage out of the things that I am being asked to do every day so that I can operate as a sort of AI powered executive? And, you know, people think this is, is far off and it can't really be done. I know CEOs right now that are building a digital twin of themselves and asking their team to ask the digital twin questions before they get asked questions.

Like, this is happening right now, this transformation about how leadership and management is happening. So I really think we're gonna have to re-skill a bunch of people. It goes back to that example we were using with the subject matter experts.

I mean, the CEO is just a type of subject matter expert. Totally. And I think a lot of those folks would just assume, refer you to their digital agent, and then if you have something more interesting, let 'em know.

Yeah. Um, when you think about all this though, I think a lot of innovation and a lot of things that we could be doing, we don't do simply because the amount of effort required to get started is too high. Mm-Hmm.

Are we gonna see a lot more innovative ideas and software? Because I might be sitting on my couch and I have an awesome idea, but you know, the time and effort it requires me to stand up the development environment, get everybody involved and get all this thing rolling. I just kind like, you know, eventually go, yeah, I'll just watch Netflix.

So, Well, I'll, I'll just, I'll speak personally. I have written more code in the last 12 months, um, for myself, for fun, for side projects, then I probably did in the five years before, because the cost of building has collapsed not just from generative AI capabilities, but I think we're seeing a whole wave of innovation and cloud hosting and, you know, serverless, all this sort of stuff that just makes it dead simple to roll out applications. I do believe the cost is going really low.

I do have this hypothesis at sort of artisanally crafted farm to table software is, is gonna have a, a movement where you can actually have very cheap micro-targeted software capabilities built for very narrow use cases. I think the question then, you know, where my mind goes is what's the actual market monetization of that? And then candidly, for the venture capital community, how does funding work in a world where it cost functionally nothing to build software?

And so I think there's just very interesting dynamics that come here, but if anybody that's watching this would like to join my club of folks that sit in front of Netflix and spin off three apps every Saturday, um, there's, there's, there's a crew of us that text each other little apps that we built. I actually, I mean, we're at the moment where this is kind of funny, you know, you used to get out of a meeting and someone would say something funny and somebody would put a meme together and like a, you know, a little thing making fun of the thing. Now truly, we have people coming outta me meetings that say something funny and they're building a 15 second app and sending it to the team as a little joke.

Like, it's that level of hyper efficiency where, you know, these throwaway apps can be as simple, you know, as, as easy to create as like a jpeg. And uh, that to me is really fascinating. So how will this all play out in your mind?

Because I could easily envision a scenario where, you know, my agent comes to the conclusion that the CEO agent is an idiot and we should just go start our own company anyway and be done with it. And we could all have our own little tiny companies. I mean, it's gonna be the, the, um, the downstream effects on licensing IP is gonna be really fascinating.

Can these agents take IP with them to other organizations? So I do think there's going to be, um, both self-regulation as vendors want companies to adopt these technologies with minimal risk to, to corporate value, as well as over overlaying regulations coming whether we want it or not. It.

So I, I don't know how to predict the future on how this, um, symbiosis between human and ENT corporation building will ultimately net out. I am generally though a optimist here. I think anything you can do to free up human creativity, um, allows us to have more impact and more fulfillment.

I would say I am both more productive and much happier when I have this set of tools next to me to take meetings off my plate and wrote tasks off my plate and allow me to spend time on the things that I think are most, um, value creating as well as the most human, you know, get to spend time with people. So I, I'm netting out on this is all positive, but I will keep you updated if, um, I start to worry about the agentic corporate revolution. All right folks, you heard it here.

The most important thing when it comes to experimentation is to remember to play. Yep. Play is how we learn and that's how we kind of make things that are fun and interesting.

Otherwise, you know, if you're thinking about AI and you have a fear and loathing, you might be missing the point. Hey Claire, thanks for being on the show. Oh, thanks for having me.

And thank you all for watching the latest episode of the Textron AI series. You can watch this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next time.