AI Platform Team with Patrick Debois at AIE 2024

Today we're gonna talk about gene AI and platform teams. I think Gene AI really deserves a platform team. Um, and I'll take you on the journey and explain why that is.

Hi, I, I'm Patrick Devo. I'm one of the co-authors of the DevOps handbook, and people kind of know me from my dev ec and ops background. I try to be the independent kind of consultant and the voice in the industry.

So, you know, whatever you see here, uh, the tools or whatever, I have no specific, uh, affiliation with them. And I try to level up engineering both on the organizational level and on the technical level, but enough about me. So I have gray hair.

And over the years there's been a fair share of new technology that came into our companies. There's been the internet, uh, I'm that old. Uh, there's been moving to the cloud, there's been mobile, you know, DevOps has itself, mini revolution, kind of how people are working agile, you know, and now the whole AI engine ai.

So how does this typically happen that it's getting introduced into a company? So first thing, people look at a use case for the new technology, and they take a simple one, maybe something they can achieve quite soon, and they kind of try to understand what it can do. So typically a one team goes in, you know, with DevOps, they call it the DevOps team.

With Agile, they call it the agile team. There's always kind of that one spark team that access a change agent for something new, and then one that, uh, gets going. Um, you know, we kind of make it work.

Uh, we, we, we, we change things in our arc. We, we connect wires where we have to, you know, we, we kind of embed the new technology for the first time into the organization, and that takes time. But we can learn a lot from that.

And that's kind of our pilot or MVP that we typically introduce, uh, with a new technology. After that, you know, we kind of learn on how we can kind of scale this out. Uh, technically we abstract things away.

So, you know, typically as we did with the cloud, with some coding, with infrastructure as code, and we build a platform Kubernetes, and it doesn't matter. So this is a pattern that we kind of abstract things way because we want to ease the burden on the developers so they don't have to learn all the nitty and gritty, but they can be productive quite fast with the new technology. And so to bring that to the teams, we find champions in those teams, the ones that are really, you know, excited about that new stuff.

And we kind of bring them the news, we feed them with information, and they kind of are our change agents that spread this across the org. What we have as a new technology. And once people start kind of scaling This out from one team to a few teams, to many teams, there's gonna be patterns of reuse that we, we start noticing, uh, we, we use it as golden paths, paved roads to kind of like bring them to multiple teams that have a standard way of working with the new technology, uh, to scale this out.

And with this new technology and there's no patterns. You know, we educate the teams to become better, to use those kind of pave roads, uh, the why they were like, uh, that in the organization, why those decisions got taken, uh, in that way. And then ultimately we keep measuring this change and the area of success, how it's being, you know, kind of spread across the whole organization.

But we put guardrails in place so people don't shoot themselves in the new foot with those kind of new technologies. You know, the, the guardrails, kind of the safety nets, the circuit breakers, just to make sure that, you know, if somebody is not well versed in the new technology, they don't do things that kind of hurt our organization. So if you look at this, uh, this is closely related to a platform team, to those kind of new to this concept that maybe come more from a data concept.

It is a way that how kind of teams started organizing themselves. First, Devon Ops were kind of like two big silos working against each other. Eventually over time, we adopted a concept of feature teams where each of the teams were kind of focused on parts of the product.

And, you know, in order to scale it out, there needed to be like a central system that put, uh, some of the reusable components in place that is sometimes to be referred to as the platform team. And the platform team then enables that. And they have like an enablement team sometimes, you know, that's the cloud team itself, or development that could be, you know, internal developer productivity or developer experience teams.

They kind of bring those new services from the platform team, uh, to the, uh, different teams in the organization. You can read more about this in the book by Matthew and Manuel on Tip Topologies. Uh, if you want to get more in depth, and there's a few ways of collaborating.

There's the actually collaborating physically and then scaling this out is sometimes, you know, brings the same service with documentation and makes it as a service. So it becomes like self-servicing to different teams. And the close collaboration is more for the exceptions in those organizations or those kind of like, uh, topics that need to be taken care of.

So the way I see it is, you know, we got new technology geni in this case, and we wanna scale this out. So for me, the parent has been clear what way we do this is, you know, when the change and the first change teams are kind of ramping up and they kind of have like a stable setup, you start thinking about a platform team. Once we build those kind of services, we wanna bring them to the different teams.

We use that enablement for that. And when we have everybody using, uh, we also put the guardrails and the rules in place to use the new technology. Sometimes referred to as governance we put in place.

Okay, first topic platform. So what goes into a gen AI platform? Well, for that, we need to know that what goes into a gen AI application.

There's some code, yes, the prompts are something new APIs we already had, and then there's some frameworks that lose this all together. And then obviously some LLM models or some, some other kind of, uh, machine learning, uh, tasks, data we bring in. And all of that together goes from the request we put in the app response with a reply.

So speaking of models, there's been over the year and a half kind of an explosion. And obviously it's even sooner those models existed. Uh, and, you know, people have a hard time choosing a model and every day there's still a new model.

So it, it is not like strange that we're struggling with, but this is probably the first request you'll get as a platform team or an infrastructure team. Can you expose some of those models? Can rerun some of those models.

And you can choose between a SaaS solution, you know, kind of like, uh, OpenAI bedrock, Google version, vertex. Uh, and you know, most people try to, in the enterprise not use OpenAI, but obviously they were the first, and sometimes they're still used. But you can also bring your own model, uh, that is not a SA provider.

And some of the solutions actually provide a uni unified interface, and they say, you can use one model, you can use another model, in this case Jurassic Claw stable diffusion, and you use kind of the same API interface to kind of like inter interact with those models. So, you know, you, you run the model, either a SaaS, uh, or one of the open sources, ones, there is a unified interface of dealing with those, and you continue to bring that, uh, into the infrastructure and there's public and open models available. So, um, a typical tendency is that, oh, we're gonna only support one model.

Reality is you're gonna have like a multitude of different models that actually fit a specific use case. You know, the purpose is important. So you kind of have to pick, is this something better for text?

Does it like deal with voice? Does it have vision? So you kind of have to understand what the use case is to actually bring the perfect model to those.

So you'll probably be dealing with a bunch or a zoo. It's sometimes called of models that you're, uh, serving up to any of your developer teams. And open source is really getting good.

So it is a viable alternative. Sometimes you want this and you can't use a public provider, uh, as the SaaS. Uh, so definitely there's solutions out there, um, that kind of get onto like the same quality of some of the closed source, uh, providers.

The next thing that you gonna have in a question for is a vector database. Now, what is a vector data? You might be new to this, think of it as an index.

Well, we know indexes from a database from a search index, but this is like all the unstructured data, PDFs, word or chats and so on. They also need to be indexed. And this has a special way of doing this.

It's called embeddings. And this kind of indexing mechanism to retrieve the right data and to index the data where it is, is called a vector database. I think there's a tendency between having a specialized vector database in the upper left quadrant or more and more.

You see kind of this solution being built in the traditional database vendors that you know, next to time series structured databases. They also have like a factor databases built in. So that's kind of emerging now over a year.

And then to get all that information indexed, sometimes you need, uh, connectors, uh, you know, whether that is your alfresco, your confluence, your Jira ticketing. There's a lot of connectors that kind of bring this in. And you'll sometimes re, uh, hear the word rag, uh, kind of retrieval augmented generation.

Basically it's fetching the data from all these sources. It understands the sources, it understands the format of the sources, and it brings that into a prom. So it's likely something you'll be opening up to, uh, to developers, whether you expose kind of the services and the data sources they're allowed to use.

And then if you want to get really fancy, um, you know, you might already get into agents as a service. You know, all of the major SaaS providers are already providing this as a service for people to run their own agents. Now, I told you that, you know, there's multiple ways of running things.

There's multiple, maybe unified way of dealing with one of the, uh, vendors, but there's also an abstraction proxy. So if that is something useful for you that you have one API and you can easily switch across model, think of it as the multi-cloud, multi-model, multivendor, you probably will have to bring in an abstraction proxy. This is also very good in a way that it kind of transforms all the outgoing, uh, requests to a single point that can be controlled from a more security point of view.

So, and that has, is also pro, uh, less prone for change or impact if one of the SaaS providers, uh, does some changes. And then obviously these things cost money. So maybe you want to implement some caching so you're not doing the same operation and query over again.

So there's caching infrastructure on top of the models and the requests you can bring in. And you know, there's obviously, you know, if you have infrastructure, you need a way of monitoring its health. And the traditional kind of, uh, providers will, uh, get you the metrics like, you know, request delay or cost.

But I, I would call these like the traditional metrics. There's much more that goes on in these kind of metrics. Uh, for the models, you know, time to first token is very similar time to first network by, um, and there's a few other like latency, throughput.

How many requests can you do things you wanna measure? It's a little bit different than API calls. It's very typical to the LLMs as they were talking.

And so you want to have like all these metrics flow to you. You know, people have been in the same thing as open teleme telemetry, have kind of like open telemetry, kind of a standard way of sending these things, uh, to your, uh, monitoring providers. And you know, if things do not go well, there's a need for tracing much like API call tracing because it rarely is one prong that goes on.

So you'll have to provide not just the infrastructure, uh, think of this as your observability stack and the developer and the debug stack, uh, that you need to provide. Uh, when people are calling these things, you don't want them to set this up. You know, per team, you offer this service, like you offer your own monitoring, uh, service to the developer teams, and then you can do evaluations.

You can check for the quality. It's not just about how fast, how costly. It's also like are they on, uh, kind of consistent?

Is it relevant? That being interest, but we'll come back to kind of those, uh, quality metrics, uh, later in the presentation. And so this continuously data quality monitoring is something you do.

Um, similar to your health checks for your APIs is something you do for your data as well. And some have started calling this rag ops if you're really into that pattern of dealing with the connectors, the data sources and so on. There's a new set of monitoring that you'll be looking at.

But it's important to know that, you know, all these kind of monitoring and metrics are important because you're dealing with production. And we always joke about like, oh, the only place you can see it in production. But, you know, we strive to do things in your test environment, but in this case, we don't know what the end user is putting in their prompts.

So we do have to monitoring monitor things quite closely in there. And besides that, we also wanna know if it doesn't work or like the API or the logic is not very real. So we capture the feedback.

Could be a simple thumbs up, could be, you know, when they click the copy code, uh, you know, try again is an indication that they didn't like the answer or kind of go even deeper, like provide them an editor so you can see what people are changing for that. So all that information you can feed back to your developer teams to understand that, you know, people are happy with it or not happy with it. And that's kind of like resulting like feedback as a service, right?

They capture all the data that you can use to kind of improve your prompts, improve the model that you use, improve your API logic for that. And then you could, things like AB testing, is this prompt working better than the other prompts? How do people like this in production?

And ultimately you can have a registry like reusable prompts so people don't have to do this over and over again in your organization. Um, next to kind of, uh, prompt, um, registries or like repos, you can also do this with your model and you register fine tune models or self-trained models in a way that people can find them inside the organization. So they're not duplicating the work there, much like Kubernetes world or DevOps world.

The tool set is ever expanding. This is just kind of like a snapshot. It will keep going.

Uh, but you know, this is kind of the level of infrastructure and platform that you'll be providing. And they're all new kind of topics of things that you're not used to. It's not just an API gateway.

It's not just kind of a, a database that you're providing. There is a new domain that requires a new infrastructure. So second part, we got the whole infrastructure running.

Well, you have not me. Um, and you know, we are enabling the teams to go and use this. So how do you go about this?

First step is you have to overcome the fear of engineers to work with ai. They typically think a, that it is about data science and it is about, you know, maths and kind of very complicated things. We, with the gen AI, are working with integrations.

So you have to make that clear. The second thing is you wanna make sure that they do not fear for their job when they're doing this. And one of the researches shows that companies that actually have a learning mentality, uh, always improving things, continuous improvements, they are better designed to deal with this AI skillset threat.

So, you know, we, we want encourage the people to work with AI and to learn more about this gen ai. And we can go, uh, not, you know, even if they're not building the application, we can get them excited, right? So we can have, uh, you know, make them examples.

We have like a hackathon, we can share links about those typical sharing of new technology, uh, and bringing that in. We can make sure that we value kind of new things, uh, as, uh, management, we encourage people to build with that. So kind of all that stuff is important to kind of make them excited about the engineering and we want them to try as much as they can, uh, without fear.

And one of the ways to do that is that you can provide playgrounds where they kind like, you know, similar to the public GPT, but more controlled, you create a playground for them to see and experiment with prompts. So that's a cool way of, you know, them learning in a safe environment without leaking any of your information. And then you want them not just, you know, type in the prompts, but connect those with maybe internal data sources that you have so they can experiment some of the logic, uh, to kind of work with those.

So the kind of, even if product owners can work with this, um, and they can think about their use cases and get excited on, you know, the new possibilities it get and also get an understanding of the limitations and where, uh, the application actually need love. And another way of doing this is that some of the developers actually like a local deaf environment. And over the years there's been kind of, uh, you know, an increased way of running a model with good quality on your laptop.

One of the ways is a LAMA or LM studio, and they will both allows you kind of like to have like a myriad of different models running on your laptop. So it is like definitely in the flow in the cycle of the developer that they can do fast, uh, prototyping if you want to, uh, in a safe environment. And eventually you kind of have them even like a visual prototyping tool where they do one step, another step and kind of go, uh, you know, you go from prompt engineering to flow engineering.

So even without having to code too much, this is something you can bring them, um, as a playground. If they really start developing anger, then, uh, there will be a discussion about should we use a framework or not. I'm a big fan of a framework, uh, you know, any of those will work.

Um, there is a problem with the frameworks is that they tend to change very fast right now. So it can be get annoying for developers, but I think like any other middleware and framework people will build solutions up on top of that. But it might be something you would have to help and kind of like educate people or, um, uh, uh, for this.

And the most important thing I think is we've been striving to write more test coverage in code that people start doing the same. You know, initially they're very excited about their use case, they wanna move this to production. And I've shown you the feedback as a service, how that is how you create a test data set, not just with kind of your kind of TDD loop, but kind of bring that in.

Uh, you know, the input and the output of that Gen AI application is something that can be tested and evaluated. So that's the term you will use. Learn them.

One of the evaluation frameworks, you know, typically you start with exact testing, maybe you can find a pattern in the output in the tech data generating, but you can also ask a helper model. One of the ones that I shown on hugging face, is this toxic? Is this relevant?

Uh, you can see how close this is in the vector database, whether that was, uh, relevant to what the question was. And you can even ask, you know, kind of an L-L-M-L-L-M as a judge. Was this a good answer?

Did it make sense in this use case? And, you know, as the ultimate thing, you can still have the manual testing if you want, but that's kind of what people are striving towards in making the test repeatable because it just doesn't scale. If you have to run through all the use cases, um, for a geni application and be aware, I told you the models are changing all the time.

The prompt is changing all the time, kind of your framework is changing all the time. So there's definitely a lot of change going on. So you have to kind of keep track of where are you using what in what prototypes.

So you know, kind of when to, uh, change this and leave in enough room and advocate that, you know, the things that they did like a couple of months ago ago, probably need some more love to kind of bring up to date with the, the new standards. So speaking of this kind of tech debt, um, will bring in the governance because, you know, that's the way we deal with kind of like legacy, um, what needs to be scrutinized, what needs to be updated, uh, in a good way. And governance can go in a variety of places.

The most important thing when people think about governance is that they don't trust the AI vendor. They don't trust about sending information to them. You know, similar actually to how it was in cloud.

So there's a, a healthy untrust, um, kind of with any of the new providers, uh, and you'll get over, have to get over it. And already the cloud know what you have. So you know, you using their services is probably what, not one of the problems, but you might kind of, uh, make sure that you trust them.

And good technology actually builds trust. So kind of like, uh, every time it works, every time it's, uh, trustworthy and you have to make sure that you understand the opt in and opt out policies in products like when they paste things in, when you use it, uh, models or services providers, or even if you expose public information, you can opt in and opt out of any of the trainer, uh, of the models itself. So be aware that kind of, you bring that knowledge about opting out in your right way.

Uh, when developers are doing some of the, the work there and you know, like software, open source libraries, if you're using one of the open source models, you have to look at licensing, is this permitted? Is this the right way? Some of those models have been actually only, uh, allowed for scientific work.

And if you use them in your kind of product, they're not the right license to you. So be aware of the licensing, much like you were doing with open source libraries there. And these models also have more detail usually.

So you look at model cards, it's a way of describing how the model was trained, what data was used, how it was optimized, so you have an understanding. So I would call that similar to, you know, the bill of materials that we are looking at in developer components, uh, and security. Um, so that gives you an understanding how it was actually dealt.

And maybe sometimes it mentioned things like, oh, there is a bias to certain things. And the bias could be en gender, uh, because it's trained on the internet and there's definitely a bias towards men. Uh, it could be racial that you wanna be aware of.

So we have to look at these model cards to understand have they've been trained, uh, and that you are avoiding any of those problems. So you cannot just have the developers pick a license just for its functionality. You have to look deeper, uh, to do some governance on that.

And then just using the models and writing the applications, um, you also have to fulfill and understand in what use case, case you are allowed to do what. There's a risk level, uh, that is indicated in European legislation. Uh, you know, depending on the risk you're taking for the human, um, you are allowed to do certain things, but you're also obligated to do certain things.

For example, show how the model was trained, how it was used, what kind of use case it is, uh, used for. So kind of be aware of that legislation that you have to comply to, uh, also for your governance there. And then you wanna make sure that whatever you build is not just untrusting your end user.

So it's sometimes referred to as a concept of humble AI where you say like, you know, the AI is here to help and not to block people for promotion or not to people on access on certain things, but it's part of the risk profile that the European legislation actually describes. And then next you wanna make sure that no private information is leaked into models and you can monitor that kind of during the training. You can monitoring that during your observability as well, that no information is leaked that goes out of the models.

And then you can start masking things when requests are being made. So just make sure that even on the sending, there is no information that leaks out. Um, and think of it this as a web application firewall, but just for input and output, your prompt and your result or your completion that you're filtering things out.

So this is a, a simple way of dealing with the kind of POI, uh, in there. And it has the similarities. It could be, you know, a simple check.

It could be a Rex, it could be, you know, much more advanced like a an LLM judging what the output is, uh, as a firewall. And you're able more and more to configure guardrails as a service where you can put like hate speech or other things that are getting blocked or your biases that actually are being blocked. So those exist as a service and as part of your governance when you kind of bring the proxy in your filtering in place.

And then for, you know, making sure that nobody is allowed to do like certain queries there should not be doing or results that they know should be gagging. There's a, you know, kind of make sure that you're protected against prompt injection, uh, similar to SQL injection. Do the right filtering, do the right monitoring and put the guardrails in place there.

And one thing I wanna highlight is that a lot of people think about is like the chat GPT and the prompt, uh, goes in and on, but there's a new kind of also emerging, which is more of a personal ai maybe, you know, in the company a lot of your people are using AI for your code generation in your sales and marketing generation, and maybe they're also pasting stuff in. So that's part of your governance as well, to make sure you understand the tools, that the tools are blessed, that, uh, your users actually are educated on the awareness when they paste things in. They should not be leaking things because it's gonna get like really crazy if your AI is just watching over your shoulders and recording everything they see on your desktop.

And we're gonna get a new set of problems, you know, likely to be solved with ai, but kind of like, you know, privacy, what gets blurred, what what kind of is not allowed to see. Um, so definitely the security and the governance is just ramping up in that space. So we got the platform, we got the enablement, we got the governance.

So my plea is start with a platform team. So where does it fit in? Before we dive into that, um, I wanna make you, um, aware of something called the AI engineer.

Think of it as the chain AI person that does not have to be a data science person, but understands integrating multiple models, multiple APIs, multiple prompts to an application. And you know, at certain point we called some people the DevOps engineers. This seems to be the industry term that we're going for, which is the AI engineering.

And the difference between a typical data science kind of data platform, uh, engineer is that they were usually a little bit on the outside. Um, they were getting the analytics, the reporting, something that is not typically used in production. There are exceptions, obviously people running models in production, but kind of the tendency was that they are much further away from production.

So the AI engineer is in a way a shift, right? Uh, not like more in development, but actually more in production, uh, towards production. So that's kind of a, a new emerging pattern in the organization.

And what we've seen is that, you know, data and engineering, sometimes they are separate in the org because of their nature, like in kind of close to production, not close to production. And we see the first data science because they're close to the models, bootstrap some engineering practices, then more engineering gets at and eventually kind of they're bolted onto the first use case and then it needs to become like a repeatable pattern. And that's where kind of the platform team comes in.

So the way that I see it is, you know, similar to the, uh, organization of the team topologies, the platform team, you know, supports all the feature teams, but next to or CloudOps or SecOps or developer experience, we have the data platform and we bring in the AI platform. And you know, some companies like Adobe, they call it the AI experiences team, the enablement team that kind of bring new technology actually to the organization. And the nice thing of putting them also in the platform team as a whole is they can learn from each other.

One team, the Dex DevSecOps team knows a lot about compliance, access control is something, you know, the CloudOps team also knows a lot about developer experiences is the frameworks and the gen AI kind of brings the, the level of models and training and all that stuff. So there's a nice collaboration, uh, bound to be happening in that kind of new cluster, um, of platform teams brought together in a single team. And your maturity level may be, you know, kind of still low, you're still implementing your first team, but we know a platform team will be leveling up and maturing people from, you know, a manual thing we did the first time to something that is documented and repeatable to something that we can automate and eventually something that is self-serviced.

You know, the other side of the spectrum is that siloed and every team kind of does it on their own, but as I mentioned, the governance actually makes sense. And the reusable components on the enablement also makes sense to bring that more central. And so for me, I'll leave you with this, the steps to make it a success, these are the three parts we learn the pattern from, you know, other platform teams.

I think this is the way to go to bring this in. Uh, maybe not. If you're like a two person company, then you're doing it all on yourself anyway.

But if you're scaling this out in a bigger org, uh, this is the thing that makes sense. And I love to talk to you, uh, if you're in your journey about this. And let me know if I can help.

You can find me on LinkedIn, uh, quite easily. Um, I'm open to any responses there. And if you wanna learn more from me, I have all my talks also on YouTube.

Find me there, like me there, subscribe me whatever you want to do. You know, there's a range from DevOps to more visual kind of, uh, learning to, uh, a whole thing of bringing things into production or demystifying more prompts. And with this, I want to thank you and you know, we're on to the next thing.

Uh, and enjoy the rest of the conference. Thank you very much.