AI Strategy in a Box – The Five Pillars of Enterprise AI Success With Bjorn Austraat at AIE 2024

Welcome to today's session on the five pillars of AI Enterprise Success. My name is Bjorn Ostra, and I've been applying AI at Enterprise Scale since before they called it ai, uh, starting with machine translation in the nineties, speech recognition in the early two thousands as part of the Watson team in 2015 through 19, and then later at Wells Fargo and Truist. Today, I wanted to cover an important part of AI enterprise success, which is to develop, uh, and bring to the table a holistic strategy that covers a lot of the pillars of AI that are critical for a successful outcome.

You may have seen stats, like 85% of AI projects fail to deliver the business outcomes that were expected or promised, or even higher failure rates on AI projects. That has basically stayed fairly constant with the arrival of generative AI as well. And you may ask yourself, with all the good intentions, a lot of executive attention with a lot of funding, why is it that so many of these projects actually never deliver?

Well, imagine if you were at a racetrack for a Formula One race and one of the teams shows up and they do their big reveal, and all you see is a race car engine sitting on a block. No, no doubt. The race car engine itself is incredibly well designed engineering, high, high tech.

But of course, you need a lot more things to successfully win a race than just the engine. You need a driver and a windshield and a pit crew and gas and tires, and of course, many of the other things to ultimately make, make it across the finish line and, and deliver the win or deliver the big tree. And in AI land, the corollary to that is that sometimes teams focus too much on the technology, or even worse just on the model, neglecting all the other drivers of success.

And then similarly, you would have a, maybe a well-designed model that performs extremely well, but at the end of the day, business outcomes rely on much, much more than just a model or just a platform. And I like to say that it's about 5% model, 95% of everything else. So what is the 95% everything else, and how can you ensure that you can deliver holistically beyond just technology, beyond just platform models and ultimately make it across the finish line to condense down the many, many complex factors?

I've created the system of the five pillars, and I'll walk you through these pillars today. And the five pillars, as you can see here are our platform solution, portfolio, communication, talent, and governance. Very often, many of the regulated industries or other folks that are applying AI are already quite good or have a lot of capability in the platform and in the governance side of the house, often also a robust talent strategy.

But I wanted to also cover the solution portfolio and communication aspects today since they're equally important. And you can think of these five pillars as multipliers. So if any of them is weak or null, then it will multiply to either a small number or small success or total failure.

So it's important to really have you see legs under you and build capability if you're not already there in all five aspects of this, not just the technology or not just the model. So let's go through them in sequence and I'll give you a little bit more color commentary about them as we go along. So starting with the platform, that's obviously a key capability.

You can think of it as like your factory assembly line for models and beyond just the model ops aspect, it's also important that you have tight integration to a data marketplace that makes it easy and straightforward and and effective to discover data assets. And I call them liquid data assets in alignment with some of the great thinking that's come out of MIT on this topic. And it's a key distinction between having data quote unquote data and having liquid data assets.

And some of the differences are that liquid data assets are discoverable, they have high quality, they have good governance and lineage, and they are also ready to use from a metadata point of view and permissions point of view, so that people can self-serve in a highly governed and, and traceable way, can self-serve these data assets. Another side topic here is synthetic data that's becoming increasingly important for a number of reasons. Privacy is one key driver of having a good synthetic data strategy.

And as you have seen recent legislation out of Europe has further emphasized the importance of very meticulously guarding the privacy of your clients, of, of any entries in your data records. And the right to the rights to be forgotten is, uh, now a critical consideration in even in model development. For example, if a individual wants to be forgotten from your training data set, what is your strategy around that to address that from a tracking governance lineage and synthetic data strategy end-to-end model ops equally important.

And that means that you should be able to readily be able to collaborate and team in the, in the process of model development, you be should be able to discover which assets already exist, and that can go from snippets of code all the way to finished models and then also which IP in both in an encoded format as in, um, for example a design pattern, but also IP in people's heads, meaning a particular skillset should be easily discoverable on the platform so that I can readily team across internal organizational boundaries, maybe even with vendors on a particular model development challenge. So the model ops platform enables a lot of that and also helps you to encode knowledge as you develop it so that when the invariable churn happens, this IP loss is minimized because you've actually captured a lot of the, the good thinking of let's say a data scientist or data engineer in the, in the platform and in the code artifacts themselves. Lastly, a key emerging requirement for model ops platforms is the ability to dynamically provision architectures either on-prem or in the cloud or in a hybrid environment.

And that means that I'm able to mix and match GPUs and CPU depending on what part of the model ops and development process I'm in, and that I am able to take, uh, an environment that's, let's say, optimized for training with a huge GPU capability and then switch and toggle that for an inferencing stage later and make it maybe a little bit more CPU heavy, depending again on the model architecture. Next, let's talk about building solution portfolios. It's, uh, a continuous challenge.

It's not tri, it's never trivial in a large enterprise. To get alignment on which things the, the design team, the data science team, the innovation team should tackle first. Building an effective solution portfolio means, uh, a number of things and it will ultimately create alignment and consensus in a non-emotional way amongst often very senior stakeholders who all think that their project should be the most important one and should be done first.

And so building that portfolio, including a, a mechanism to commonly define what, uh, what things mean, what is important, what makes something a good AI idea, is absolutely critical. I call this moving the innovation left because, uh, it's true that, uh, subject matter experts, let's say in a bank, the mortgage experts or the wealth advisor experts have a tremendous amount of subject matter expertise that they need to bring to the table, and they need to pair that with an effective way to distinguish between good and bad AI ideas. And there are bad AI ideas, meaning things that should not be done with AI that may be done with something, uh, like robotic process automation or organization change management and things like that.

And so introducing a common language and a common framework that can readily be taught to subject matter experts is a, a great step towards moving innovation left. You equip people with what I call the minimum viable education to distinguish between good and bad AI ideas or fit or not a fit for ai. And so that helps you bring better ideas to the, to the start starting line of the intake process.

You'll get higher quality ideas at higher velocity, ultimately improving the quality of your innovation funnel for ai. Design thinking is critical in this context, meaning that you consider the outcomes and the user experience and the proper framing of the solution overall. So you don't fall into the trap of, you know, AI experts, I'm a hammer, everything's a nail, I'm an AI expert, everything's an AI problem, which of course it isn't.

And, and actually very often the second best answer is no, we shouldn't do this with AI or machine learning. We should use another technique or something entirely non-technical such as a reorganization or organization change management, a process we design to address, let's say a a a common challenge to the enterprise. Ultimately, it's not only important to sift the ideas properly and help you sequence them into a, a meaningful roadmap, but it's also important to distinguish whether you should buy or build or buy and build, because it's almost always a blend and distinguish between what makes a good AI project idea for internal production, internal processing, and where you should partner.

One way to do that is to think about your ability to execute in a two by two as one axis and then the impact on the other axis. And that lets you see where you should place your bets and where you should partner and where you should try to pursue this development yourself. If you have a high impact, but low ability to execute, then that probably lends itself for partner organization or maybe even an off the shelf, uh, software solution that you can just purchase.

And if it's something that you are very comfortable you can do and has high impact, uh, but you are not certain if the of the outcome that then lends itself to a fast win, fast fail quadrant where you should, you can readily experiment and see if the, if the ultimate impact is what, what you and your internal customers expected. So I like to call this creating elegant simplicity. Uh, there's in large enterprises typically no shortage of ideas, but it's also important to properly prioritize those ideas and to, uh, create alignment in a non-emotional way, as I mentioned amongst folks so that they can see, well, you know, this is a, this makes a good AI idea.

We all agree why, or this one is maybe a lower priority. This is not quite suitable to this design pattern using ai, gen ai, et cetera. We should de we should deprioritize this one, or it has a bigger dependency downstream.

And so we should think about it in context of the overall, overall ability to roll this stuff out because there are downstream consuming systems, let's say, that are on a different timeline and require additional planning for integration. Communication is one that I find is often overlooked, and it's criti, it's absolutely critical. One, a number of drivers make it even more important in AI land than in traditional enterprise deployments is because the technology's moving so quickly and there's so much, uh, i I would, you know, call it, uh, things that people don't know they don't know.

And then there's also very firm misconceptions. So you'll have seen, you know, overly optimistic, uh, uh, abilities of ai. This is becoming self-aware and it's, you know, like commander data from start, from enterprise or it's a terrible machine like the Terminator.

And in reality, of course, AI is, uh, just math. The current state of AI is still very far away from general ai, artificial general intelligence, and it's a, it's a very sophisticated tool that's sometimes appropriate, but also it's absolutely critical to have consistent and clear communication to create alignment across all levels of the, of the corporate hierarchy and across all levels of, of technical competence and insight. So the, the, the one camp of, uh, you know, data scientists, PhD, great data scientists who develop amazing algorithms and the, and at the absolute cutting edge of ai, then there may be senior leaders in the, or the board members that are familiar with ai, but not very technical, but they are very familiar with corporate strategy, business strategy, implementation, how to roll this stuff out and make it work in the bigger competitive context.

And there's of course, many, many moving parts inside any organization, startup or global 50 company. So how do you, how do you create clear and compelling communication is I already mentioned it's useful to have a common framework to define terms so that everybody's on the same page. What drives success?

One possible framework is the four Ds that I developed, uh, several years ago, and the four Ds stand for data dollars differentiation and deployability encapsulating the four major drivers of, uh, deployability and on of enterprise success. And by giving folks this common vocabulary, it's possible to not only move innovation left, but also to en further enable a smooth communication as you go further down in the project and as you execute, because, uh, people will then more readily be able to communicate and say, oh, this one, you know, has a low four D score, or we, we, we realize that the deployability is low, and we can thus create a common vision that is anchored on common terminology, a common framework, common ontology, and also equally important to, to phrase the outcomes in a way that is accessible to as broad of a stakeholder set as possible to create this framing. I like to ask a simple question at the beginning of any project, AI or not, and that framing is, or that question is, whose life is going to get better by how much and how do you know that?

And it's a surprisingly simple question with a surprisingly complex answer, often because for that you need to consider the end user outcomes, whose life is gonna get better, what will that actually look and feel like? And uh, how do you know that means? How, how will I instrument the solution?

How will I prove that the AI has delivered, let's say quantitative provable gains? And there are ways to go about that, uh, that, that have to do with a top down approach. So we, we think that we are gonna gain 2% market share, a bottoms up approach, such as we've automated two minutes away from 10,000 calls a day, that's a savings of 20,000 minutes at a dollar a minute, and so on and so forth.

So the, the how do you know that also critically all should include the voice of a senior executive who will stand up for the solution and will say, yes, this is worth 10 million, $20 million of savings per year, and here's where I will account for that in my budget. And then it all at the end comes together. By answering that question comprehensively, you will, uh, be able to create the common vision that everybody's becomes a touchpoint, a touchstone for everybody, and also readily communicate to other people that are not part of your project team.

What are you up to? And, uh, within this context, it's also important to consider, uh, what I call cognitive courtesy. That is the, the consideration of what will be the most effective form of communication in the context of the audience and the overall communication situation.

Technical folks, doctors, data scientists, engineers naturally form a shorthand language that is effective amongst themselves. Of course, it becomes highly ineffective and inefficient when speaking to folks that are not sharing the, the technical education and skill that is present. And you may have experienced this yourself when a doctor maybe gets suddenly too technical in their explanation of what's going on, and they have to translate this now into a language that is understandable and effective amongst folks that are not, did not go to med school.

The same is true for data scientists, car mechanics, and so on and so forth. Importantly, it's very, very important that you consider this framing that it's about translation, not dumbing it down, because if you think of it as dumbing it down immediately, it's amongst unequal people while translation is among peers. So thinking about cognitive courtesy as pret translating your message and your outcomes and your impact to the respective audience so that when the communication situation arises, you are ready to speak in a, in an accessible and memorable and credible language to that different stakeholder set.

Communication can also, uh, happen in non-traditional ways, and that communication means to the outside world to to communicate what it is that your organization is up to and not just say that you're innovative or that you are, that you are interested in engaging, let's say Gen Z or Gen Alpha in your AI journey. And, uh, one of the innovative forms of doing that is, for example, to host a competition where you use probably synthetic data to, uh, have a competition. The famous platforms of Kaggle, there are other platforms that do this, and then teams from around the world can compete on solving it is science problem.

Not only will you be able to gather ip, but also you will open your talent pipeline and communicate effectively through action to the broader world that it is what it is that you're doing with AI and what it is that you see, how you see yourself as an organization, as an innovator and an engaged member of the data science community. I already mentioned talent and this ties in with your talent pipeline. Um, when you introduce AI and now more, more recently, generative ai, it's critical to think about the upskilling and, and the, the other related incentive alignment methods that have to be in place to bring along the team on this new technology journey.

So incentive alignment can mean many different things. Uh, that is, let's say the bonus payout, how that's driven for everybody in the company. Should it have an AI component that could be, I've taken an introductory class to AI or I've built my own model, or depending on the role, of course, we'll have many different shapes that this will take.

But if you really want to use AI pervasively in your organization, it should go beyond the pockets of data science and engineering and data engineering and so on and should really touch everybody's life. One great way to do that is to make this part of your goals, whether they're incented or not, uh, and your performance reviews that that happen at some regular interval in your organization. And make sure that everybody is really coming along with this, with this journey and has a reason to do so beyond the natural instinct to want to educate yourself and learn more about AI as it is a hot topic.

And, you know, generally, of course will be relevant to most professions going forward. Cross-training is another aspect that I've heavily emphasized in my teams, and that means that, uh, for example, if you are a PhD data scientist and excellent already at the algorithm portion, I want you to cross train pitching and, and making a project pitch or even a project phase pitch that will work well, that will extend cognitive courtesy to the outside world. And instead of talking about, let's say dramatization or doing hyper parameter tuning, which is not effective when speaking to a senior executive, you think about how you could phrase this, this activity, the things that you've been working on, and make it effective and memorable and accessible and credible to an outside, to an outside, uh, member of the broader team of the organization or even somebody you made at a conference.

TA knowledge capture is critical given the typically high churn of data, data science and data engineering resources. And that means that I'm able to, in a consistent fashion, exfiltrate and document what is in people's heads. Uh, one novel approach with this that is broadly enabled using generative AI techniques is, for example, to create podcasts or video interviews of senior stakeholders across the hierarchy, technical and non-technical.

And then using generative ai, speech recognition and sentiment detection and the deep understanding of the transcription that LLMs can provide and use that to create almost like a chat bot of the collective brain of the organization. This is now possible with relatively little effort and was used to be much, much more difficult with traditional natural language processing and can give you a novel and innovative and an innovative way to capture the tacit knowledge that's in the heads of seasoned folks and unlock that value to the rest of the organization. Whether it's for onboarding or for, uh, ongoing training, sometimes snackable training, uh, that means I can do it in small increments with, uh, the rest of the, the team.

The communication also means external talent pipeline generation through being present at conferences by hosting, for example, competitions that I mentioned previously, right? And so you can see that these various branches of this strategy are actually all interlocking. com.

So if you want to download the visual representation, including some of these cross connections, please do so after this conference. And then last but not least, AI has to be ethical. It has to be safe, has to be considerate of rules, regulations, and your own moral and eth ethical obligations to your team and your customers.

So a, a comprehensive governance framework is absolutely critical to execute AI safely, to stay out of regulatory trouble and to simply live up to your own expectations as to what this technology should do and what the real world outcomes and impacts should be on your clients and teams and the broader society as a whole. When looking at your governance framework, there are some key requirements that must be present, such as the ability to safely handle data, to, to allow for the right to be forgotten, for example, in data privacy, to manage and retain the data correctly and to account for data movements and data use to the broadest possible stakeholder group, including the regulators. When it comes to models, it's important to have proper model documentation and have the right degree of model explainability and interpretability, and most certainly have proper monitoring after the model goes live to ensure that you can detect drift.

And that drift can be in many different forms. It can be a privacy drift, it can be accuracy, drift, performance drift, and so on. And as the model is live and its training is complete, invariably it will start to drift because the data doesn't stand still.

And so a, a delta will open up between the training data of the model and the actual real world data for which monitoring is absolutely critical, and that you can also detect if something goes out of bounds and suddenly your credit scoring model, for example, starts treating certain demographics according to protected lines differently. That then requires you to either retrain the model or take some other corrective action to stay aligned with your ethical obligations. Adversarial robustness is another core consideration.

Models are software at the end of the day, and they can be attacked. You may have seen attacks on generative AI projects or generative AI endpoints where people give the model adversarial prompts to make it do things that the designers and users and owners of the model did not intend to happen or did not intend to, to expose. And so the robustness of the model against such attacks has to be a key consideration to also remain safe and hence have good governance across the entire model ops lifecycle from beginning data engineering to model development to operationalization, and then monitoring.

And then lastly, thinking about generative ai. Some of the larger providers such as, uh, Claude or OpenAI Chat, GT four and other models like that are behaving really a lot more like enterprise software than traditional models in the old days. Amazingly, you know, a few years ago when you would build, let's say, a fraud model based on gradient boosting, you would be, you would be able to explain the entire model document, every part, every use case, every potential outcome was testable, end-to-end testable.

And you would of course also know the training data, the type of model interpretability and explainability of something like a large language model like G PT four is quite profoundly different. It behaves a lot more like enterprise software, hence less interpretability, less insight into the actual functioning of the model. And also less frankly, understanding of even the makers of these models, what all the edge cases can be.

So robust testing becomes more important. And an integration of this test cycle, continuous daily, weekly testing of the model's performance of its behavior, uh, becomes critical. And in this way it behaves a lot more like enterprise software than a, a traditional, let's say, gradient boosting model to detect, um, fraud in a, in a credit card processing model.

And, uh, thinking about ai, gen AI as part of A-T-D-L-C technology development lifecycle or a software development lifecycle can be very helpful because a lot of the patterns are actually quite applicable in this case. So with that, we've covered a ton of ground. If you'd like to find out more or read some of the postings and, and ongoing insights and articles that I continuously publish on LinkedIn or in my substack, please connect.

I look forward to hearing from you, and I hope you enjoyed this overview of the five pillars of AI enterprise success. Thank you.