AI/ML Platform Engineering with Chris Corriere at Techstrong Con 2024

Hello, my name's Chris Coyer with Inspire 11, and I'm here today to talk to you about AI and ML platform engineering. I, So about me. Um, I am a associate technological mathematician.

Uh, I have a background in complexity science, uh, and have been a consultant tech for over 20 years. I'm also an organizer for DevOps Days Atlanta. We're back at the Historic Academy of Medicine, uh, this October.

Uh, and I am also organized the only serverless days in Mapc camp, uh, in Atlanta. If you're familiar with wardly mapping, done a lot of wardly mapping work. My time won't be discussing that today.

Uh, and also a lot of value stream mapping, which we will be getting into some value stream mapping during this talk. Uh, I also relate to Dr. Strange, uh, it's a Arthur C.

Clark quote that sufficiently advanced technology is indistinguishable from magic. Um, I have some chronic pain issues and have also spent a lot of time, uh, in studying practice, uh, to, to adapt, uh, to those issues and, and figure out how to get ahead a little farther with tech. And a shout out to my employer, inspire 11.

Um, we do a lot of work in the AI space. Uh, so yeah, moving on. So starting off, what is a platform?

Uh, this usually gets hammered down into the specific cloud context these days. Uh, to zoom out on that from a sociotechnical perspective, the two examples I like to give are an airport in a, a shopping mall. So in Atlanta, we have Hartsville Jackson.

Maybe you've passed through, uh, on your way somewhere else. Maybe you stopped in for a while. If you visited Atlanta, maybe you've been through Lennox Square Mall.

Went there a lot growing up. Uh, but a broader definition as a platform is a two-sided network. Sometimes you have more than two players, uh, provide opportunities for businesses to serve customers.

So you may not give money directly to Hartsfield Jackson when you go there, or Lenox Square Mall when you're shopping. Uh, but they provide lights, parking, uh, running water, uh, and spaces for businesses to come in and rent that they can do business, but without the overhead. A lot of the operational overhead that is common among the vendors.

Um, and that's where this concept of platform comes in, that these common, uh, things like power, uh, maybe gas, uh, running water lights, all the KTLO trash service, for instance, uh, that those get abstracted out where the individual vendors, uh, folks using the platform don't have to worry about them. Um, and your cloud platform is really trying to do the same thing. Maybe you're on AWS, maybe you're on Azure, maybe you're on GCP, uh, but you're really trying to abstract the broader, uh, cloud services that are available along with other third parties you might be leveraging, um, and abstract that for the teams in your organization.

So they don't have to deal with all that complexity. They can focus more on getting their application running and launched. Um, starting with, uh, a get commit.

Um, something to note here is quality of platform service and, and what's the expectation? Uh, so self-service versus concierge, uh, self-service is usually cheaper and gonna be more general on what's available as we move into a more concierge specialized service. There's a luxury element that comes along with that.

Um, from traditional cloud compute into AI and ML platform, um, it is a bit more specialized concierge service. Um, and that's because it's still relatively new. Uh, as you practice this within an organization, you get your platform team set up, uh, you're probably going to have folks embedded at a team level when you're smaller, and then that platform team emerges over time.

We're seeing the same thing as we shift to AI and ML practices, DataOps practices to support, um, at a platform level that we're having to partner, uh, with more of the AI ML cloud at a team level. Um, but as we get into a rhythm, uh, there's some cross pollinization and we pick up techniques from each other, uh, we're able to abstract it out. Things get a little bit more cheaper and become a little bit more self-service.

So, uh, pipelines is value streams. This is more the current state. Uh, how we're dealing with, uh, web compute, um, per door report is, uh, keeps us in a complicated space where we have repeatable experiments.

If we start with code commit versus, uh, story origination, um, that pull request starts a, a build pipeline where we run into static analysis, uh, during build and unit testing, uh, which gets us through our first quality gate. Uh, assuming all the unit tests pass and the links don't catch anything, uh, that artifact is gonna get us stored in a binary re repository, which allows us to build once and deploy to multiple locations consistently. Um, and finally that's gonna hit our first lifecycle, which is where we're able to conduct initial runtime analysis.

Uh, so on the other side of that, we typically see, uh, these four environments in some version or another where you've got smoke testing or, uh, dev dev, your first lifecycle, um, where developers can confirm the application launches and really just stands up and runs as expected. Quality assurance is gonna get more into automated integration testing. Uh, there should be some manual exploratory testing, um, being completed by test engineers there.

And then, uh, finally, I like to see a pre-production gate. Uh, sometimes this is called staging instead. Uh, but this is a gated environment.

You really need to know what applications are running simultaneously to recreate prod like conditions. If you're doing performance testing, there's other dynamic analysis that can be conducted in this lifecycle. Uh, typically this is where I like to run, uh, z attack proxy or burp suite or other security analysis tools.

Um, if you throw integration tests at that pre-production environment, it replicates some level of user activity, gets memory seated with things that would be populated if actual traffic was on, um, and do that before that dynamic security analysis. Usually get better results. Just a little tip there.

Then finally, if everything checks out in pre-prod, uh, we move to production and can observe our application under real user traffic, um, and see what users are actually up to. Of course, in some more modern environments that move to production is just moving traffic to a non-prod environment versus moving the artifact to, uh, a completely different, uh, location within the cloud. Um, shifting points of constraint, uh, this quote from Einstein came up that if I had an hour to solve a problem and my life depended on it, I would spend the first 55 minutes describing the issue to raise the proper question, uh, before spend the last five minutes solving it.

Unfortunately, I only have 30 minutes today. Uh, so I can't spend the first 55 minutes on anything. Uh, but I did want to just run into moving from traditional compute where the, the point of constraint has shifted, uh, from DevOps and SRE, how platform engineering, um, is having to adopt to AI and ML moving forward.

And this is just more field notes. So, uh, take these with a grain of salt. Uh, so the point of constraint is now IO not compute.

Um, I actually heard Josh Long Starbucks man speak about this for the first time, uh, with respect to, uh, reactive spring, but we're in the past, we were able to scale compute, uh, in production both horizontally and, and vertically, where we were less concerned with, uh, memory and CPU utilization. 'cause you could always throw more compute at it. It's easy to scale that up and down.

Um, where our bottleneck shifted, where the point of constraint is now is around IO and moving data. Uh, so a lot of these AI ML applications at various points between models, uh, data that they're ingesting, training data they need, uh, you're looking at terabytes of information. Um, it is a lot of data management.

Uh, and that is the compute is only gonna help you so much if you're at network saturation. Um, so this is where async communication has become necessary. Um, you've got RPC functionality in projects like, uh, Java springing and Apache Arrow.

There's a lot of neat stuff going on with Apache Arrow. I don't have time to deep dive into either of these today, but something you may want to take a look at, uh, in your neck of the woods problem too. AI and ML technologists are a little bit different.

Isn't that true of all of us? Um, so really smart subject matter experts I've run into in this space. I use the word technologist here.

Uh, 'cause sometimes it's developers, sometimes it's systems administrators, uh, but the culture around the organization they're in, if it is an AI ML company, there's a, a certain perspective on tech. Um, you're gonna run into a lot of Python notebooks. Uh, unfortunately, these don't test or scan exactly like, uh, traditional Python files do.

They're, uh, the Python notebook format. It's technically A-J-S-O-N doc. Um, so there's some jumping around we can do with, uh, converting them to traditional Python scripts, uh, to set up testing and run them through a linter.

Uh, it's some feedback. The the line numbers don't always match up exactly. Um, but it's better than nothing.

Um, it's not a perfectly solved problem at this point, in my opinion. Uh, not only are these artifacts different that, that we are getting, um, where we could abstract a lot of this stuff. Uh, you can say something like, well, you just have to submit a PR into the source control repo, uh, that we're running out of, and I'll be able to, to package this stuff up.

Uh, a data scientist may not be familiar with GI on a command line or, uh, infrastructure's code or what Kubernetes is. Um, and you know, really what the difference is between a Python notebook, um, and a traditional Python file, uh, eager to learn. Um, these are all things we can get through.

Uh, but it's, there may be some, uh, cross pollinization that needs to happen where your traditional DevOps crowd, cloud infra crowd learning more about Python notebooks and things like Databricks. Uh, and on the other hand, the, the AI ML crowd may be picking up more cloud infra, uh, and things like command line experience, um, as is space matures. Problem three, um, Python is slow.

So this, uh, PDF I've got a link to here, uh, is a research on efficiency of programming languages across energy, time, and memory. You can see Python is pretty low on the list here. Um, I love Python.

I, I, it is my first go-to, there's a point with projects where performance, we're serious about performance, we're gonna have to port it to Russ, Java or c plus plus. You can see those are at the top of the energy and timeless for the most part. Uh, Java doesn't do quite so well on memory management, which is understandable with respect to JVM and some other things.

Um, but in terms of energy and time, uh, Python isn't particularly environmentally friendly. On that note, it is gonna generate more CO2 as a result of some of this inefficiency, but it's also gonna impact your end users. A lot of the time.

This performance issue isn't a problem. Um, if this is an internal application with maybe a hundred users or less, uh, it doesn't need to be super fast. Uh, and another business context, if it's, if we're trying to hit global scale, we've got users on, uh, all the continents, then uh, maybe we're gonna run into a problem that we can't just scale up with more compute to fix again, because we're, we're at an IO constraint.

Um, this is a rough gen AIOps architecture diagram. Uh, so at the top, we've got, uh, document management for vector storage. This vectorization is really the big shift here.

This is not so much a statistical model, um, with AI and ML anymore. It's more of a vector space. Uh, the mathematics seemed to translate pretty clean.

If you've got a background with vectors from, from college or personal study, uh, if not, something to look into. What forms of basis of a vector space and how distances are calculated is literally what these vector stores are doing, um, within this middle tier and this, uh, model life cycle. You've got foundational models, uh, which you're gonna run more locally.

Uh, think LAMA tube versus monolithic models, uh, like, uh, GPT from open ai. Um, this is where you're gonna get into data governance, uh, making sure that things are clean. There's room for model training, um, validation testing, and fine tuning.

Uh, then you're gonna have, uh, this third party data, um, which is more player on the field data. What's in your vector store should really be like what's on the box, top of the game. So the rules, things should be running by versus players on the field.

Um, then that final step on the operational lifecycle is, uh, you've got gen AI tracking, logging, and monitoring mix of experts where you're pulling in multiple models to get some form of consensus, uh, to make sure it's correct. Um, there's also room to pull in LLM audit and critique, uh, in the, the same vein where you, you've got something being intentionally critical, uh, and an interest of improving quality. Uh, there's also caching and gating if you, the same prompts are getting hit repeatedly.

Uh, there's room to improve performance and cut some of your cloud spend, uh, with intelligent CLA caching here. And finally, your UX UI step at the bottom. Again, Python notebooks are really popular in this space.

It does not always make sense, uh, to cut, uh, gen AI application and launch it into the web, uh, on its own where it's got direct access to users. Uh, keeping a human in the loop is still very popular in this space. Uh, so how production ready can Python notebooks be?

Probably they're, they're running in, more of them are running in pro today than you would probably, than I suspected at least. Uh, but it seems to be best solution for where we're at, uh, in the industry right now. So I, I converted that diagram into a bit of a value stream for Gen i a, uh, rag applications.

So RAG is retrieval augmented generation, which is demo. I'm gonna run here quickly if, if we've got time. So, uh, at the left we've got our data.

That corpus of data, rules of engagement, those get stored as vectors for similarity search. Um, those are gonna get bounced off our model. Um, we have a model strategy there, foundational or monolithic.

Do we wanna fine tune? Is there any quantization? Uh, which is another way to affect performance?

How precise do we want our model to be? And is it worth the cost of that? Um, then at this prompting stent, uh, we have lang chain.

There are other options available in that space. Now, uh, LAMA Index comes to mind, um, this context orchestration and our middleware application code. Um, some folks believe this Lang chain, uh, middleware piece is gonna go way over time.

Um, it is still entirely valid approach. Um, I'm still using it, obviously. Uh, then we get our data integration, uh, this player on the fields data.

So what contextual data can we provide it? Um, there is an operational cycle to consider here. So looking at vulnerabilities, for instance, from five years to present day versus the last six months is a different report that is a different context window to he and an LLM.

And it's gonna impact the responses. It gives you, not one is objectively better than the other, but depending on the answer, the question you're trying to answer, you're gonna need to change the data and make sure you're not handing it anything stale. Um, the more relevant it is, the less sifting it has to do on it.

Its end the better. Um, and finally, in the observability space, um, are we gonna live broadcast this thing or keep it offline? Uh, do we want to air gapped?

If we're running a foundational model, we don't have to have this hooked to the internet at all. Um, there are some folks with interest in doing that. Um, and again, we get back to mixa experts audit, critique and cache.

Uh, so generally setting one of these applications up, this is the flow I'm moving for. As you can see, the conversation is starting with data more so than compute, um, as it has in the past five, 10 years. Um, so this is gonna install some dependencies and give us a hell of a low world outta OpenAI.

Uh, some of these are already satisfied on my system, but I'll, I'll go ahead and run 'em anyway. So this is the Lang Chain OpenAI model. Uh, once we've got that, I've got my environment file over here, that's got my API key and some other stuff in it.

I'm not gonna open that up. Um, you may need to run this dot environment package to, to load your open API key. Once that's complete, um, we should be able to, to kick off a simple hello world.

You can see it handed us some code here. Uh, and we can try this example a few times to see it's not consistent. So, uh, an LLM is gonna hand you a response back.

Um, and, but we're dealing with a complex system, not a complicated one. So these experiments aren't automatically repeatable. Part of the challenge moving from traditional compute to, to, uh, this gen AI space is, uh, testing it.

It needs to be similar enough, um, but it's not necessarily supposed to be identical every time. Here we set temperature, uh, which helps with this. So if your temperature is cold, um, it's gonna be more serious and try to reply more.

Literally. Um, if the temperature's warmer, it's gonna be more creative. So if you're asking it, uh, something you know about a legal document, you would want the temperature to be cold.

'cause you want it to be very literal. Um, if you're getting it to help an LLN to help you with creative writing, maybe you want to turn up the temperature so it gets a little, uh, embellishes a little bit more. Um, so you can also stream results by, uh, character here.

I'm asking, uh, where in the world is John Willis? Uh, John's been, I've been participating with Sean in a number of j AI hackathons. Uh, some of this, this notebook actually is based on a repo.

Patrick Deo launched. Um, he is Jedi forever on GitHub, if you wanna find that repo. A lot of this demo is based off of that work.

Um, we got to collaborate together on some of this. Uh, John Patrick and I both in Boca last year and then up in New York in January. John is super active and always on the move.

So where in the world could he be today? Honestly, don't know. Um, and then it tells us, John's a renowned speaker and author.

Uh, he is got scheduled speaking engagements probably in the United States, if not on Twitter. Uh, various tech conferences. Could be San Francisco, New York, or Austin.

You get the picture. He's back in Atlanta for DevOps days, Atlanta this October. So I will see him then again, if not sooner.

Moving on, uh, we're gonna set up a simple open AI chat bot. Um, so installing a lang chain dependency here, and I've got this prompt set up where we've got, we're telling it, it's a helpful assistant that provides security advice, and we're gonna ask it what makes DevOps secure? And so it gives us a few ways it can contribute Automation.

We're familiar with collaboration. That's correct. Uh, ci i CD is important from a platform perspective.

Uh, IAC so this is a pretty good answer, but this is just what, uh, open AI has been trained on, what our l LM has been trained on. So for next step, um, I guess I've got this as in a template where we've got it asking a joke. So tell me a joke about what, and I can give it a funny joke about ai and it's going to respond that why did the robot go on a diet?

Because it wanted to reduce its bite size. Um, that's better. Some, some of the other jokes I've gotten from open ai.

Uh, so in a similar template, um, we can also introduce it as a, a chat prompt, uh, which is more optimized for conversations. Here. You can see it is, knows its name is Bob.

And we've got some information on, uh, our tokens and what the prompt consisted of. So on the document loading, um, you can see I've got a PDF folder here. I've got the CSA roadmap for ai, the NIST AI guidance, um, uh, o osp, uh, application security verification standard.

And a puppet stated the DevOps report all in PDF format. Um, shout out to Tracy for providing these, uh, she was, uh, down in Boca with us and had helped with a lot of the security guidance, um, and data governance pieces in this space. So, um, we're gonna install PI PDF to help ingest these.

So here, um, I've ingested the, the NIST document. Um, we've split the pages with the loader, and here I'm grabbing page two. And you can see here, it gives me the page content from that PDF.

And if we scroll all the way over, um, it tells us in the, the metadata what the source document is, what page number is. We can also just load all of these PDFs from this directory at once. This one does take a second.

It is loading a lot of data. Again, if this was in rest, maybe it would be faster. I, so just to talk a little bit about chunking and splitting while that's running, um, we, we've gotta break these documents into pieces, which is what the chunks are.

And then after we split into those chunks, we have to consider the overlap. So for a level of continuity you want each year, if your chunks are separated, it may be missing part of that inner context. So you've gotta, that's why it's called lang chain, is you're chaining the language together, uh, to provide it.

That context, uh, tweaking your chunk size and your overlap, um, is gonna affect the performance of your doc. Uh, PDF is one thing. Uh, markdown chunks very, it's easy to split markdown based off a header, for instance.

Um, depending on the document you're loading, um, you may need a different splitting and chunking strategy. There's a ton of documentation and examples on Ling chain's website. And this should be wrapping up here shortly.

So now that we have our PDFs loaded, um, we can go ahead and run, uh, this chunk and split. So looking at this is the 10th index split. Uh, in this list, you can see the page content, and then if we scroll all the way to the right, we can see that this is at out of the NIST document.

This is page five. We get a start index. Uh, so now we've gotta get this, um, embedded into our, our vector store.

This is actually just running through embeddings. Uh, so each of these got embedded. Then if we run through what this and the, it gives us the size of the first embedding, number of embeddings calculated.

Then if we print the second embedding, this is actually what the vector output looks like. This is the number of parameters. So you're talking end dimensional space is how I've been referring to this.

Just an idea what this, this data looks like once we get it embedded into a vector. Um, CHR ADB is a vector store, so, uh, you can install that with pip And Then, uh, we're going to set our DB up with those documents, all the splits, uh, with open AI as our embedding model. And now we can run a similarity search on those documents.

So if I ask it, which documents have something about AI in 'em, you can see we get a number of responses. Uh, most of these are out of the CSO roadmap. Uh, if we were to give it DevOps as a search term instead, Now we've got mostly results from our puppet state of the DevOps report.

So, uh, moving on to our final step, we're in the home stretch here. What we're gonna do is take this contextual data in our Vector store where we've got Chroma DB doing similarity search. Um, and we're gonna set up a prompt where we can pass our question through similarity search to pick up contextual data that we've provided from the PDFs, and then pass that on to OpenAI to answer the question in context, not with just the data it's been trained on, but with data from the PDFs provided.

And this is at the heart of a retrieval augmented generation or what they call QA rag. Uh, so you can see here, I've got my prompt template set up. Uh, use the following pieces of context to answer the question.

If you don't know, just say so don't make up an answer. Then we provide it in the context and the question. Um, then within the docs, we've got formatted, we passing in our retriever here, which is based off of the similarity search out of, um, R DB or Chroma db.

And then we've passing through that question. And the question we're handing in again, is what we asked gen, uh, AI earlier, is what makes DevOps secure? And it says, according to the context, good security practices, uh, better security outcomes are enabled by DevOps practices.

And I have this pulled up, uh, in the search already. Uh, and you can see that this is straight from the context of the state of, uh, DevOps report. Uh, so instead of giving us a generic answer off the data it was trained with, it found a contextual answer based off the sources we provided.

Uh, and this is probably the quickest, uh, return on investment I've seen with Jenna at this point, is, uh, getting your data into a Vector store and being able to get, have it answer questions, uh, contextually. So next steps with this, um, is how our rag bot my answer questions more accurately than just our vector similarity search loan. And then again, how we could split and chunk this data, uh, for more effective results share.

Again, we've got John Willis at the Academy of Medicine. You can use Techstrong for 10% off code. Our theme this year is, uh, do ops stemming and generative ai.

Uh, shout out to Techstrong for pulling our group together back in, uh, August. I wouldn't have a lot of this content without that hackathon session and, uh, their continued support for DevOps days Atlanta in the community. Again, I'm Chris Corer with Inspire 11.

Feel free to check out our website. If there is a job opening that looks interesting and there's something we might be able to help your organization out with, please do get in touch. Thanks.