Operationalizing AI: A CTO’s Field Report with Damon Edwards & Jody Mulkey at AIE 2024

Hello everybody. Welcome to a, uh, special session we have here, um, in the a IE conference. Uh, we're talking to, uh, Jody Bulky, who is a, uh, well, I'll describe you as a veteran, uh, CTO, uh, Shopzilla, Ticketmaster.

GoodRx. Now First American Financial. Um, how's it going, Jody?

It's going great. Thanks Damon, for having me. Yeah.

So the point of this conversation today is, you know, um, you've been working for a lot of, uh, consumer oriented, um, uh, you know, uh, companies for years now. So, you know, I know you've had to manage teams and entire organizations that depend a lot on machine learning and, and, uh, other types, I guess we'll call legacy AI now, right? Um, but now this, you know, general AI is burst on the scene.

I know you're jumping into that as well. So, um, the point of today is just kind of give your, your field report, right. Just what's, uh, what's been going on for you and, um, you know, what you're seeing happening and compare and contrast to, uh, how it was in the, uh, in the past.

So I think where I'd like to start is, um, you know, kind of do a little compare and contrast here, right? Talk, you know, think about how life was before managing machine learning in a, in ai, and then since generative AI has burst onto the scenes, um, what's different, what's the same, what's different kind of how are you thinking about, uh, things? Yeah.

Well, um, I think about the differences, to be honest, is mainly in the, um, in the early days of machine learning at scale. Yeah, yeah. Uh, there you really had to spend a lot of time on the infrastructure, and it was very kind of bottoms up.

You spent so much time, you know, uh, not even just the data wrangling, but literally the infrastructure and kind of building models, um, and really trying to just bring that to life. I mean, back in Shop Silla, we wrote our own search engine because things like solar and lucine hadn't been invented yet. Mm-Hmm.

And, you know, so vector data, try building a vector database in 2003, not, not so easy, right. And, um, you know, you know, there's plenty of tools for co-sign similarity now, right? Super relevant.

But back then, you know, to judge the relevance of, uh, search engine results, we had to write our own co-sign similarity. And so I think that what you get to just start on top, like, you should get to start with so much leverage. I think the other thing is that the ability to pivot and go down new pathways is probably a hundred x compared to what it was even 10 years ago, even five years ago.

And so, I'll give you an example. Um, at GoodRx we built an amazing kind of marketing model, um, to help optimize marketing across like 14,000 variables. And the team comes to me and says, Hey, we found a new data set for another 3000 variables.

Um, we should have that model up and running in the next two, three weeks. So that was like four or five years ago. Um, try doing that even five years before that, that would be a six month project.

Yeah, right. And so it's just, I would say that the ability to be able to take in more, uh, more data, uh, to be able to explore more avenues, to find more insights and more opportunities, it's just accelerating incredibly. And with Gen ai, I feel like that goes on steroids.

And so, uh, I think it's to, as machine learning these other things, it's very structured problems. Whereas I feel like gen AI helps you attack these unstructured problems. Um, so a lot, a lot more free for, that's what I would say.

Um, I would think of the things that are the same is like getting it to work at scale in production, like confidently, that's still hard. You know, people are still working that out, you know? Right.

It's not, you don't see that many in the wild. You see a lot of toys, you see a lot of cool demos. You see a lot of very interesting, um, ideas.

Um, but you at least I haven't seen many, you know, large scale production things that are out there, you know, serving customers, driving revenue. I haven't seen that quite yet. Yeah.

Well, I definitely wanna come back to that, uh, to that point. But going on this kind of, you know, transition here that you've seen, like what, was there any moment in time or certain use cases or, I'm just kind of thinking how, you know, in your mind, w when did it first sort of hit you that like, oh, this is different, right? The generated AI is a, uh, you know, a, uh, a big leap forward in what is possible, or maybe it's felt gradual.

I'm just kind of curious, what was your personal experience here and what kind of caught your, your mind? Yeah. Well, I mean, I, I think when, you know, what was it the fall of 22 when Chet GPT was born?

Yeah. Is that right? Yeah.

Um, exactly. I would say that that was a, like an incredible oh wow. Moment.

Like, this is nothing like I've ever seen before. Right? Uh, and so I think that was kind of, at least for me, from a technology, a pure technology perspective, I'm like, wow, this is a thing.

This is a, a totally different, uh, thing and opportunity. And then to see that evolve into the multimodal models that are out there now with video and images and audio, it's just sort of mind blowing. And even like this week with the release of check GPT for O for Omni is, you know, it's, um, it's unthinkable.

And so I think what I've seen is that this moment, so it's been in only, you know, the last 18 months. And then I would say that the thing that's most interesting and what's different is that the rate of evolution is increasing what feels like exponentially. I mean, humans, technology's been increasing exponentially ever since the first human got a stick and knocked down the apple from the tree, right?

Yeah. And we've been using these tools to build better tools to solve problems faster. Mm-Hmm.

And now the power of those tools that we're building, it's very meta, the tools can build their own tools to solve these problems. And so it is just the rate of innovation is, is sort of, you know, unseen. I think it's unseen.

Yeah. So now you've, uh, I've been known for working at some pretty innovative, uh, companies, you know, first American, right? It's, you know, it's the largest, what largest title insurance company in the United States processes everyone's mortgages.

Like it's kinda one of these old, you know, kind of cogs in the, in the heart of the us um, economy, but it doesn't feel like a technology forward type, you know, type company. Right. Um, I, I dunno what you can share, but can you share a little bit about sort of what, you know from your, with putting your first American hat on, kind of what catch what is First American's interest in generative ai?

Or what are some of the ways that you're thinking about how these technologies can be applied to, I mean, something's been around for a hundred plus years, right? Yeah. Well, I mean, I've been at the company 13 months and, um, you know, one thing that was made clear to me is that, you know, in the seventies, the typewriter was a big innovation, right?

It's literally like things were handwritten before, right? Yeah. Or, and like, uh, things microfiche, all of these different kind of ways to store property information.

Um, but, you know, for us, you know, first American has a big history, uh, of, in, uh, innovation. They've been a leader, you know, sort of since the inception. And, you know, over the last five years, they've made significant technology investments in a digital native settlement platform, uh, to drive escrow Mm-Hmm.

And also in an automated underwriting or purchase transactions. And so we've been investing in that for the last, you know, probably five years or so. Uh, and really doubling down on the innovation piece.

And I think for me, what drew me there was this opportunity to help transform, um, such a huge part of the US economy. Yeah. And so, you know, real estate represents trillions of dollars of transactions and, you know, these mortgage backed securities, which are really the second largest, um, uh, security out there kind of class of securities, uh, really drive a lot of the, uh, economics of the United States.

And so to be able to work on something to make that better, that's incredible. I think there's just tons of opportunity. For me, what drew me from the consumer background was that probably a child and buy closing on a house are like the two most stressful events of someone's life, right?

Hmm. And, um, that whole process of buying a house and closing on a piece of property, whether you're selling or buying, is hugely fraught with anxiety. And, you know, we believe that we can help make that a smoother, less anxious process by driving clarity and communication and really helping people understand what's gonna happen, uh, making sure the process is smooth, flushing out any issues early on in the process so that it's not a fire drill in the last 12 hours of a, of a deal.

Yeah. And so, like me, there's a big opportunity just from a consumer perspective, like if I could put a, uh, apple watch on every home purchaser's, uh, arm and look at the average heart rate decrease over time, uh, through using and technology, I would be pretty stoked. That's funny.

That's a good, that's a good, that's a great way to look at it. So, you know, you mentioned, uh, the GPT-4 Oh, right. And like, just so many cool, just cool demos, right?

And like, but it's all you said, it's like you can't just do technology. That's cool, right? You gotta have business cases, you gotta make an ROI, you gotta understand like, you know, how are you investing for the better the betterment of the, uh, first American shareholders?

So great to know, like, kinda more broadly, like how do you think about investment in this AI technology? Like, how are you thinking about prioritization, you know, value return, um, you know, when somebody approaches you in your organization with some idea, like how do you evaluate it? Just kinda curious if we can talk a bit about how you think about, you know, turn this into real business value.

Yeah. Well, I mean, we've had a pretty clear strategy. That's one of the things that attracted me to the company, that the strategy has been the same over the last five or six years, and just executing and making progress on the strategy.

So, you know, for us, it's an accelerant for our strategy. Um, specifically we're investing, uh, like I said, in the digital settlement platform. And so a digital native, uh, part of this process, if you've ever bought a home in the United States, you'll know that it's still a lot of paper involved.

Uh, and so there's a lot of players in the ecosystem that could benefit from digitization. Yeah. Uh, and then also in the automated underwriting for purchase transactions.

And so, you know, for us, you know, we have a vision, uh, that the, we should be able to have a title report on the shelf for every property that could be sold. And so, um, as you can imagine, uh, it's a pretty large undertaking. We're dealing with imperfect data at the county level, 3,400 somewhat counties, all doing it differently, um, um, with varying degrees of data quality.

Oftentimes what, when we go through the title process, we find errors, uh, and we correct those errors in the public data. And so I think for us, we see the opportunity and to really help accelerate our ambitions to automate that process, uh, to drive, uh, more efficient transactions. And for us, you know, uh, like at the highest level, not to oversimplify, but we're a documents and workflow company.

So on the document side, like we are assembling documents, um, and we are combing through them, like looking for challenges, right? To issue a title policy. And so, uh, gen AI tools are very helpful in that arena.

So we have, uh, a few different projects kind of all under the banner of improving the automation, uh, for underwriting. And, uh, and so I think that's like, we're just doubling down on those pieces. Um, and then on the digital settlement platform, you know, uh, if you've ever bought a piece of property, whether it's a residential commercial, you know that there's a ton of back and forth, uh, that goes between all the parties.

Mm-Hmm. And as an escrow company, we are set to be the quarterback of that transaction. And so we have visibility into all the parties, which is, you think about that in the real estate industry, we're the central nervous system.

And what I think is that being able to give context and real time intelligence about what's going on inside of a transaction, being able to look at the components of a transaction and be able to score it, uh, from a risk perspective of, is this deal gonna close on time or not? Hmm. Um, and then use AI agents to go and surface out the background, needed to solve some of those problems early on in the process.

You know, we have a unique advantage in that we sit on, you know, millions and millions of transactions over the years. And, um, you know, we believe in a future where we can use the history of those transactions to improve the quality throughput, uh, and efficiency of processing those transactions going forward. Interesting.

So, I, I, I'm assuming it's gonna be the same way in your organization, but you know, a lot of companies I talk to, uh, it feels like, you know, this is a brand new area, right? Using AI to solve these business problems. So, so many of the great ideas come from the troops, right?

They bubble up from, from kind of nowhere. 'cause it's just like a brand new skillset, brand new way of looking at the world. And people are like, come with something that just, it's different, right?

Yeah. So, given that so many of these ideas now are, are coming up from, you know, from, uh, uh, you know, from the keyboard layer, right? We'll, we'll call it.

Do you have any advice for folks out there who are in those roles closer to the keyboard engineers, you know, individual contributors who have great ideas? Like how, how do you, you know, from your perspective where you sit, what's the best way to get those noticed? Or what's the best way to kind of package or think about, you know, for them to help, you know, forward the ideas that, uh, maybe in the past they weren't really the ones coming up with the ideas, right?

Yeah. You know, I think that's a bigger problem than just ai, to be honest, especially in larger companies. Um, and so my advice is maybe tailored a little bit to ai, but it's more, uh, it, it, it, it, it's for so much more.

Yeah. And so what I have found to be, uh, lemme describe a couple of failure patterns, first of what not to do. Sure.

Hey, there's this great new technology. It does, uh, this acronym better than this other acronym, and creates this amazing set of things that business people don't care about. Like, so I would say don't, don't do that.

Don't do that. I would say is that, um, I would frame the benefit, um, and I would speak in the currency of the stakeholders. And you know, in most companies, dollars and cents are like at the top of the list.

Um, and in many of them it's customer experience, and many of them it's competitive advantage. Um, it's really kind of level set that and tune that to the company that you're in. But it's really about framing up the problems in, uh, like that solve a, a customer problem, creating a business benefit in the form of, um, you know, more revenue, more margin, uh, better reputation, you know, um, et cetera.

And so that's kind of why I think that where I would take it. The other I think is that, uh, we really have to be sensitive around the, uh, data privacy and security concerns regarding ai. It's my belief that, you know, the, the beginning kind of, uh, the, um, the first couple of pitches, 'cause I think we're probably in the second inning, maybe in the, uh, we're in the beginning of the second inning of the game of the, of this modern AI game.

The first couple of pitches, um, really made it harder for companies to adopt because a lot of the challenges, uh, were kind of surfaced and how this just treats, uh, data differently. And it's really hard for, uh, you know, the legal department or the security department to know how to, how do I treat this thing? It's scary.

And to them scary means default, no. Right? And so I think that it's really important to address, you know, hey, we're, we're partnering with a reputable third party, right?

Um, they're not going to use our data. And we have these controls around protecting our customer's data and our data. Uh, and I don't think that you have to compromise on any of those things at all.

Um, but I think you probably need to lead with those to kind of get the, to even have that idea heard. And so that's what, that would be my advice is like, take the data security privacy issues off the table so that the person you're talking to can actually hear what you're saying and is open to exploring this idea. Gotcha.

So it's interesting. It's like, it's like another set of concerns. You gotta shift left, right.

Spend this time shifting, left testing and security right now it's like, hey, shift left the compliance, uh, and privacy concerns because you know what's gonna happen. So solve those early in the pipeline, Solve those super early in the pipeline. And then what we are doing, right.

Um, we're kind of in the early stages of this is really rethinking, you know, uh, like every company has, um, you know, I don't think they call it AI governance, but definitely like model governance, right? Yeah. So there's a lot around that, um, around using predictive models and, uh, making sure that they, you know, don't have bias and all, all of these different things.

So that's a highly regulated, highly compliant every, you know, company, uh, has those, we have those, we've had them for years. But I think evolving that to the AI world, um, is a little bit different. Um, and I think it has some different concerns.

And so the other piece that we're doing, which I've been doing in software for, let's just say a long time, I'm feeling old these days, um, for a long time, is really we're investing in building a platform to abstract those concerns from the day-to-Day Development Act activities. Because it is not reasonable to think that Heather a, uh, software engineer, uh, with four years of experience is gonna be an expert in all these things. Like the system needs to handle those concerns.

Yeah. Because it's not reasonable to put that load on all of the individual, uh, kind of contributors. Yeah.

And so we're looking to encapsulate those requirements into the platform itself. That makes, that makes sense. That makes sense.

So sticking with the engineers here, um, you know, just curious, what, what, what do you feel like the reception's been amongst engineers? Do you sense excitement? Do you sense trepidation?

Do you sense some combination? Kind of any, you know, I Think it's all over the place. And I think the drivers of it are, you know, there's like engineer engineers who are like, this is incredible.

I mean, this is truly incredible. You really get like, wow, this is fundamentally different. Um, and this provides me capabilities that I could have never had before.

Um, I think you get engineers that are like, wow. Uh, they're on the FOMO train. They're like, I'm hearing all about this.

I need to go learn this. And, you know, they're, they're getting their way through it, but it's not coming from the themselves having that moment. Right.

Sort of. And then I think that, you know, there's some are like, oh, I don't believe this, this is a toy. You know, this is not really super useful.

Um, I'll give you an example. We have, uh, been using GitHub's copilot since last July, and we've had steady month over month growth. Um, uh, the team at GitHub has given us access to some advanced analytics.

Um, but what's fascinating to me is that, um, it, it takes, what it looks like in the cohorts is that there's a certain group that they use the tool for a bit, and then they have a step function growth in their usage. And so, if you remember, um, Twitter had like the 30, like follow 30 people kind of hook Mm-Hmm. That once Twitter users followed 30 people, they were stuck on the platform.

Or once, uh, I think Facebook was 10 friends Mm-Hmm. Right. Once you got 10 friends, you were just addicted to the platform.

So I think that, at least for us, the co-pilot, my, uh, gi up co-pilot world is that we're trying to get enough positive interactions where people will give it that next level of investment in learning. Interesting. And then they can't live without it.

Yeah. And so I think that there's that, you know, we haven't figured out what that magic number is or what that kind of pinball metric is, um, for them to get hooked on it and demand that it exists. Um, but, um, but we're kind of, again, continue to kind of work on that.

Um, also fascinating on that is that, you know, our acceptance rate for copilot recommendations is pretty much stuck. We moved it from like 25% to like 28 to 29%. Mm-Hmm.

What's fascinating on the weekend, it goes to 45%. And so, uh, it's like I, maybe it's a bias towards the folks engineers that are using it on the weekend, but I also saw another public research paper, which had a similar percentage gain in acceptance rate on the weekend. So there's something about working on the weekend that, uh, drives higher or better prompting and better prompting drives higher acceptance rate of the recommended code changes.

Interesting. What, what, what kind of pro product, I mean, I don't have exact numbers, but what kind of productivity gains would, would you, would you wage would you wager that you're seeing amongst, you know, engineers or teams that have kind of wholeheartedly jumped into Yeah. You know, a copilot style of working versus those that maybe just kind of dabble in toe?

I Think it's hard, you know, we're struggling to manage it, um, to manage, like really try to understand that we're trying to like, okay, how go, where's the business case that shows that we're 30% more productive? Right. And so I think that what the way that I look at it is that we have seen our throughput velocity, so our cycle time improve Yeah.

Um, while raising the bar on the quality of code. And so, uh, in the last six months, we have, um, like many companies have done, uh, we've instrumented, uh, quality thresholds in the code, um, through the build process. And so we've raised the bar on what it takes to be able to check in your change.

Um, and we've given the team tools to help automate that, that method. So a lot of our unit tests are written by copilot. We has, uh, historically we're in a unit testing kind of place, and now we are religious about unit testing.

And so, uh, copilot in that use case fantastic. To help us. The other, is it, like every other company, we have a ton of legacy code Mm-Hmm.

And to be able to highlight a code block and like explain, explain what this does to me, super helpful. Right. Right.

And so those are the base cases for, uh, you know, copilot, you know, we're looking, we're working through the process to partner with the company, uh, to do code translation. And code translation tools have been around forever. Mm-Hmm.

Um, but now there's, uh, like gen ai, LM based code translation tools to help you migrate a platform from, uh, an old unsupported framework that's hard to get account for to a new modern framework that's supported and has a good, uh, supply chain for talent. Hmm. That's Really interesting.

For me, that's an amazingly obvious use case. I'm surpris that more, more folks are not going after that. And, uh, we're excited to kind of, the, the early signs on that are like, wow, we could have spent two years rewriting this app, or we could spend, you know, four to five months, uh, writing this app with, uh, a, uh, gen AI engineer, uh, as a partner.

That's incredible. Um, so now, how are you thinking about the organization in general, right? Like in terms of, you know, are you building AI teams that support other teams?

Are you trying to get everybody going? Like, what the investment, like what, what, what, what is this, what's the, how is this getting into your organization and how you're driving it from a kind of tructure? Oh, I, I laugh because we have this debate.

There's, it might even be an open email thread on my computer right this moment about this very topic. And so kinda how I see it, um, is, uh, and this sort of came together, my thinking of it kind of came together in the last couple of months, which is, we're, we're so early in the technology iteration of this that I think the more e experimentation Mm-Hmm. Like the better.

So I, like we are a hyper federated company, uh, meaning like, our organizations are like really, um, independent in what they do Mm-Hmm. And so I believe that's playing at an advantage for us, uh, on some of these experimental technologies. You know, we're trying to do a better job of coordinating so that we, uh, all aren't starting building from the scratch that maybe we can lay out the cover of the puzzle box and divide and conquer on different pieces of the puzzle.

I'll give you a great example. In our company, we must have 20 rag implementations, right. On like five different platforms.

And, um, but no one has built a human feedback loop. And so it's like, Hey, why don't you not build more on your rag platform because that's a commodity now, and someone take on building a human feedback mechanism, right. To make it better.

And so we're trying to get a little bit more coordinated on that, but I think that you'll see this kind of divergence of like experimentation and then a consolidation right. Into some core building blocks. So, I mean, it's kinda like the wardly map in action, right?

Right. Is like, and these level. And so I, I like this parallel, um, learning because we're getting more surface area on the learning itself.

So I really like that. I'm trying to coordinate them more. Uh, and, uh, our, uh, Paul Hurst, who is our chief innovation officer, runs the venture fund and helps shepherd some of these innovation efforts.

We were gonna, uh, put a group together to coordinate and collaborate across the company. Uh, and so that's getting, uh, we're getting some more traction on that. But again, it's moving so fast, it is moving so fast that it is short circuiting processes in large companies.

Right. And so imagine if you take three to four months to approve a piece of technology, well, that's not even a relevant technology anymore, Right? Yeah.

Right. So on that note about the speed of technology, um, you know, the types of acqui technology requiring, whether it's open source, whether it's vendors, like how are you seeing that play out? Because like, you know, can, vendors can't move as fast as the open source, some can, um, you know, some things you need to own, like as a skill internally, some things you like, obviously wanna bring external skills in.

Like what, what are your thoughts around the ai, because I'm sure you get a lot of pitches, right? What are your thoughts around the, the landscape, what the landscape, what it, what it, what it means to you and how you're handling and what you're seeing happen? Yeah.

Um, I sort of operate on the assumption that much like, uh, rolling your own infrastructure that like all the model building will get commoditized Mm-Hmm. That I basically just go, like, that's my basic assumption is that it's really expensive. Billions of dollars to train a model, like a large scale like native model, like no one's gonna do.

Right? Um, I mean, there's gonna be 20 companies that could do that. Maybe a hundred companies in the world, right?

And so, um, a company like ours has no bill business building a basic model, right? Mm-Hmm. Um, and so like for me, we're relying on kind of, you know, the kind of three major cloud providers, right?

And sort of betting that they are gonna figure this out and offer that as some level of service. And for us, what we're focusing on, what are we always gonna be in the business of? We're gonna be always be in the business of creating title products.

We're gonna be always in the business of close helping our customers, whether they be direct customers or B2B or our partner, uh, in the banking industry. We're gonna like, help them close deals. And so we're really focused kind of in that lens on the technology that helps that, uh, solve that.

And so underwriting, as you can imagine, it's a very esoteric part of the law. It's very specific. We are a company built on edge cases.

Um, you know, title insurance is the only type of insurance that actually, um, insures the past. It doesn't insure the future. And so we have incredibly low, um, you know, we do a lot of work to make sure that the past, that we can, um, assert that and provide you, uh, insurance that this title, uh, is yours.

And if there's a and any issue, we'll handle it. And so for us, it's like we are gonna bet that from a technology perspective, that the cloud providers are gonna continue to do that, and it's gonna continue to go up the stack. Like for us, bedrock is a great place to start, like in the AWS world, right?

But let's start at the bedrock level so that we can go across different models. Yeah. Like ultimately given the business opportunity, it may be worth it to basically run to do an ensemble approach to solving some of these problems.

You know, if the, um, if the business value at a per transaction level is thousands of dollars, then you could probably afford to run it against five or six models, right? Yeah. To do a good answer, right?

Uh, and so I think that we're gonna use them as a, the kind of base and then count on their abstraction to be able to handle some of the innovation, like hide the model improvements from us. Yeah. We don't want to do ation, et cetera.

I think as you kind of go further up the stack, you know, um, if you wanna do it today, you have to build more of your own. And so I think that, but over time, it will continue to go up the stack. We'll be more and more, right now you have pretty fine grain services.

You're gonna get to more course grain services. Um, and so we're just betting on, you know, that's where it's gonna be in the next two, three years. That's great.

Hey, I know we're running out of time here. Um, tight, tight deadlines, but, uh, I wanna thank you for, for joining us. Uh, so it's some great insights.

I think, you know, people, it's really a service people to hear from other folks that, to know, Hey, am I crazy? Am I the only one feeling this way? Or, what's this journey, uh, look like?

It's so, uh, it's so new out there. So, um, I wanna thank you for, for, uh, for joining us today. Excellent.

My pleasure. And you know, it, it, it's a huge opportunity in front of us. Yeah.

And it's gonna be the folks that have experience who've seen, who can do some pattern matching on prior kind of parts of the technology evolution, and then who have really clear domain expertise in their world to be able to see, uh, their problem space, like with gen AI in their field of view. And so I think that, like, that skill, being able to see the problem space and knowing the tools and knowing the sorry tools will be too low of a level, knowing the technology Yeah. Um, is really gonna be a differentiator for folks in their career going forward.

Um, and I think that that's just a great place to be. So no better way to learn how to do that than to jump into the arena and start iterating. Yeah.

Think you said it well, that's a great place to end it. All right, Jody Mulkey, thank you very much. Thanks.

Take care.