The New AI Stack: From DevOps to AIOps – Technologies for Enterprises in 2023 | DevOps Experience 2023

Hello, my name is Mark Henkel, and today I'm gonna be talking about the new AI stack from DevSecOps to AI for ops. Um, what I do every day is help people use and understand artificial intelligence. Uh, I welcome you to follow me on Twitter.

io. Um, and I am a full-time artificial intelligence expert and consultant for Perty Labs. So, all right, let's start with artificial intelligence.

I'm not gonna tell you about artificial intelligence, but we all see it in all our news. All of our work life is being permeated by ai. The thing that's really interesting for us that are operators is that artificial intelligence is bringing a new wave of apps, and these apps are sometimes referred to as stacks.

Um, this is a legacy term. Um, we had the lamp stack in the early two thousands, which was Linux. Um, my S Q l Python, p h p, and Apache Web Servers.

We use this term. And this term is actually comes from Alan Turing back in the early days of ai. And the stack really in, um, implies that there is a linear stack of technology that has an input and an output.

Um, while that term is still used, it's not as relevant today as it was in the days, um, that Alan came up with it. Today. When we talk about stacks, what we're really talking about is a group of tools that delivers some kind of solution.

And let's just look at a couple of ways that we describe stacks. Sometimes they're just in an ontology, which just a list of different, um, infrastructure and application terms that are all logically grouped. Um, sometimes it looks like a process.

This is a process of what the new, um, AI stack looks like from the venture capitalist Andreessen Horowitz. And here's another stack, yet again from another venture capitalist Sequoia. And they are logically grouping certain technologies together.

Um, but at the end of the day, what we're talking about is a loosely coupled group of technologies that produce some output. And I actually think they look more like a web. And so I'm not gonna call them the new AI webs, but basically, infrastructure is, um, connected.

And rather than being in linearly connected, they're loosely coupled together, they're microservices. And those services that are coupled together actually become the infrastructure that serves ai. So rather than talk about stacks and the frameworks that we're used to of layered stacks, I'm gonna talk today about a simplified framework for AIOps.

And that simplified framework is not necessarily the, um, logical connection or, uh, technical connection, but actually the categorization and the groupings that allow us as existing operators and DevSecOps and to understand that infrastructure and apply the tools and the processes that we already, um, know and use. So this is what my new AI stack looks like. And you can see that I've, I've grouped things by infrastructure.

Those are the stack side, the, the things that provide AI services, um, the way we, we, uh, deliver, um, chatbots and other vision models, things like that. And the other side is the AI operations side, and that's the side that looks very similar to what DevSecOps are already familiar with. I, um, the C I C D, uh, monitoring and observability, um, configuration management and security.

And while this doesn't necessarily show how they're interconnected, it does show how they're related. And so, um, at the center of this, um, what I'm calling my new AI stack is a enterprise large language model. And that let enterprise large language language model would be the equivalent of chat G P T that's specific to an enterprise.

It may be through their enterprise offering, it may be through Microsoft Azure. It may be provided by another provider. But all of these things somehow, I believe, are gonna be related to a sort of enterprise l l m large language model.

Or they're going to be using a large language model that is publicly available, but has certain enterprise security controls. So if I take my nice pretty, uh, pie graph and, and overlay my web, it looks something like this. So the idea of the web is that some of these things are connected throughout the circle and travel across these networks to connect.

Um, and they're not exactly a one-to-one relationship. They're all networked together. So let's start with the brain.

And that's the enterprise, L L M. And I mentioned that it is going to be a central repository of data that's either served by a public model, like open, open ai, or it's gonna be a trained custom FA foundation model, like something you'd find on the website hugging space, which is, um, in essence the GitHub for, um, AI and machine learning. So you would use one of these two models that provides, you know, broad, large language model, um, capabilities, you know, content generation, um, semantics, um, uh, summarization, things like that.

Inference from, um, from its training to new problems. That's the center of the large of the new AI stack. Now, let's look at the infrastructure side of the equation.

Um, so the infrastructure side, I have tried to group these by four logical groups. We have our data, which is the data from our enterprise that goes into our AI infrastructure, fine tuning and training. That's the way we take the data and actually, um, adjust the model to provide inputs or provide outputs that are customized to our use cases.

There's narrow ai. And narrow AI is, is basically a specific, uh, language model that has a specific task. So it could be a virtual assistant, it could be a diagnostic assistant that looks at, um, x-rays in a hospital based on vision data, and, um, helps to diagnose those, um, x-rays, MRIs, that kind of thing.

And then there's model integration frameworks. And the model integration frameworks are really the middleware of the cloud. So let's just start with the, um, data layer.

And this is the things that we already have in our enterprise. We have data warehouses, we have data pipelines. Those things are streaming data all day long.

And the thing that we're probably going to need to add to that data model is provenance and tracking. So we want to know when that data enters our large language model so that we can, um, know how it's trained, what kind of data it's making decisions on, and know that, um, where our data goes. 'cause a lot of times these large language models are black box.

We're, um, it's probably one of the hottest topics today is are we, uh, maintaining the privacy and security of our data when we interact with, um, some of these large language models? That's probably one of the most, um, um, compelling reasons why people do not use AI today is, especially in businesses, they have privacy, uh, and regulatory concerns. So we have our data, we have that coming in.

We're probably storing that in something called a vector database where we're converting, um, data that might be in the form of words and numbers into vectors. And vectors are, um, mathematical representations of that data. And it allows that data to be analyzed by algorithms like a nearest neighbor algorithm to find semantic, um, similarities and group that data closer together so that it can, um, search through it through vector search and, um, do lookups much more quickly.

Okay, once you have the data into the model, um, one thing that a lot of, um, organizations will do will be fine tuning. And so fine tuning is taking a model and adding, um, context from your data. Um, you may take that data and label it on a dedicated data labeling platform like, uh, um, human signals, uh, label studio or label box.

And those data labeling platforms allow humans to actually, um, tag that data and tell the machine learning algorithms how to interpret that data so that it can learn. And then those machine learning backends actually process that on GPUs and TPUs, um, which are tensor processing units for those that aren't familiar, um, which is a chip set from Google. And what they do is they, um, learn from that data so that they have a corpus of knowledge so that they can infer when they, um, encounter new types of data.

So if we saw, um, pictures of cows before we ever saw a cow, by the time we went to the country and saw a cow, we would understand how that works. That's the fine tuning and training. Fine tuning is less of a commitment.

Training is a bigger commitment. So fine tuning is dating and training is marriage. And training is, uh, a much more resource intensive, uh, process.

Mainly what we would see from um, companies is they will probably fine tune something called a foundation model, which is one of these models that's been trained by, um, companies like Facebook. Um, they have their llama too, or other models that are out there and freely available in hugging face. And then we will just tune those models 'cause that training data they used is pretty broad and provides a good understanding of things.

Now, the last part that I've, I've grouped, or the next part I grouped together is actually what I call narrow AI and autonomous AI agents. So narrow AI is very much the same as your large language model, except it's geared towards a very specific TA set of tasks. So, um, I said earlier it could be a virtual assistant, it could be a content writer, it could be a scheduler, um, it could be any kind of, of task-based narrow ai.

It could be, um, trained on data from the x-rays, like I said before. And its specialization would be to look at those x-rays and based on the data it was trained on, um, determine, um, what the diagnosis for those x-rays would be. The other thing is related, and this is the autonomous AI agent.

So these are agents that are actually, um, goal-oriented. So they're, they're almost the, uh, they have a set of tasks and a set of goals that they can continue to iterate on until they reach the goal. So if that goal is to plan your itinerary on a business trip, you could say, these are the dates that I'm going from North Carolina to California, and I like to stay at Marriotts in San Francisco.

It would use that data to actually work its way down through, um, that list until it achieved its goal. And there's a couple projects out there like Baby A g I and Auto G P T that are sort of proof of concept projects, but that's probably not the, uh, long-term solution. There will be autonomous AI agents that are task-based that we'll use to help, um, take the menial work out of our everyday life and offload it to artificial intelligence.

Then the final grouping for my, uh, AI infrastructure are really what I call the middleware for, um, AI infrastructure. And there are two main projects out there. There'll be many more, and there are others right now.

But, um, they're called Lang Chain and Lava Index, and basically their frameworks for helping to build, um, AI applications. Uh, you know, they're very, very complex. Um, they take the ability for you to chain together conversations across models and, uh, in the case of Lang chain, um, and provide other tools for your, um, app billing, AI enabled applications.

Now, let's flip the coin and we're gonna go to AI for ops. So if you've been in DevSecOps, um, and notice I didn't say just DevOps, but DevSecOps because I think that security is probably one of the most, um, important things for us to talk about as we we, uh, move through this AI adoption curve in the enterprise. So the DevSecOps folks, we have sort of the, um, three main areas of management.

We have the C I C D monitoring and observability, observ observability and configuration management along with security. And I call this side of the, the coin, the AI for ops side. So operators who are looking to, um, take care of AI infrastructure, they can map their existing systems to AI by using this framework.

Um, and we can start with continuous integration. And so, uh, continuous integration and deployment, this is really the, the same processes that we're familiar with when we're deploying other large infrastructure in the cloud or on-prem, um, except what we're integrating now or in infrastructure that is specifically for serving ai. So we have that model training that we talked about earlier.

Um, that training data will be now be part of the configure continuous integration or fine tuning data. Um, we'll want to test, so just like you would run, uh, tests against your builds, you will also wanna run tests to make sure that, um, the model has been updated and it's not providing information that is not relevant or biased. And we'll have a set of tests for there, um, whilst have version control.

So if we do update the model and it's not performing as we like, we can roll back. Um, same thing on the continuous deployment side. We'll, um, deploy the models just like we deployed other infrastructure while the ability to scale that infrastructure.

So, um, these large language models may scale out as, uh, um, have different processes, and that'll be done by, handled by continuous deployment. And then we'll have the monitoring and feedback loop. So if there's a failure, um, you should be able to automate the, the rollback based on some kind of failure coming from, um, your monitoring.

Now, let's go to monitoring and observability. Um, so monitoring and observability is, you know, a standard thing that we do with our technologies today. Um, monitoring is very much about, you know, performance and health checks, resource utilization, um, all those things that you would do for a web service or email or any of your other infrastructure.

And then you go to observability. And observability is probably a newer trend within systems management, and that's giving us insights into how the model is performing, um, uh, why it produces this particular output, and gives you the ability to trace through, um, these processes. And if it observes, uh, you know, erratically, you would be able to find the root cause using observability metrics.

And then we have configuration management. So most of us are familiar with configuration management tools, um, from HashiCorp or Puppet or Chef. Um, the configuration ta uh, management tools that we use for AI are gonna have, uh, maybe the same, they may be new, but they will have all of these metrics that we're, we're looking at our confi, our parameters rather.

And that is how you would be updating, you'd be up updating things for model versions for training, configurations for deployments. Um, all of these, these things, you'll have the same process, the same tooling and processing in concept, but you'll have a lot more, um, different infrastructure that you have to manage in typically the same way as you always have. And last, but certainly not least, you have security.

And so the, the big concern from a security standpoint is, you know, is your data staying secure when you enter it into these models? Um, is the model, um, adverse to attacks? Um, is there a chance that the inputs are poisoning the model?

Um, so model poisoning, same as data poisoning. Um, when you deployment, deploy it, is it as secure as any other web service, et cetera. So all of these things are considerations now that we're in the, uh, AI infrastructure world.

It's just the, um, attack face is different and additive to what you already have. So now we talked about, you know, what the infrastructure is and how we would manage it, and we're gonna start talking a little bit about the metrics. So the metrics around it, I think are gonna be very much what we see already in the machine learning and the ml, uh, ops field.

So we're gonna want to know accuracy, um, which is self-explanatory. Data drift is, um, over time as the model, um, ages, um, the data can actually shift the results one way or another. So we want to be testing for that through some set of benchmarks.

Um, uh, explainability inference time, uh, model inference. All these things are, um, explainability is a metric around, uh, um, qualitative and model inference time, uh, is quantitative, uh, training time, uh, the amount of time it takes to train a model, uh, precision. All of these things are, um, numerical and can be, um, quantified easily.

Um, some of these more qualitative metrics are a matter of doing tests. And then once you get into the, uh, AI observability, you'll start monitoring these monitors for, um, models for uptime and system health just like you would from any other system that you're, you're managing. So key takeaways from today.

Um, AI is bringing a lot of new infrastructure to bear. And while that infrastructure is different, there's still some similarities to what we're used to in the past. Um, you know, stacks are a terrible descriptor in my opinion.

Um, I think webs are better for the cloud error. Um, and having a framework that allows us to have a conversation for existing DevSecOps professionals, I think that's important. Um, the tools, I think the category categories are the same, but the way we apply them are slightly different and we will see a lot more metrics.

And these metrics are, um, not quite as black and white as, um, uptime and, um, uh, performance. Some of these things are much more complex and have a, a more of a background around machine learning ops. So, um, there will be some learning curve.

But all in all, I think the, uh, technology brings a lot of positive things to bear. And if you have the framework to discuss it, I think that makes things easier. So thank you for your time today.

Um, again, my name is Mark Hinkle. Um, I provide a weekly newsletter on, um, trends in AI in the enterprise. io.

So I'd love to see you subscribe and thank you to for showing up today.