GenAI for DevOps with MongoDB with Darshana Paithankar at AIE 2024

Hello everyone, and welcome to the session on Gen AI for DevOps with MongoDB. I'm Darna by Tinker, and I lead a product marketing team focused on DevOps, security and IT ops here at MongoDB. And super excited to share some of the, uh, insights here on MongoDB and how you can use it to create your Gen AI applications.

So, let's get started. You know, in today's session, we'll be starting with a quick introduction of, uh, gen AI setting the context with, uh, rag vector search, uh, and all the things that you need, uh, to get started with your Gen AI models. We'll then look at how you could use MongoDB for your Gen AI applications.

And then we'll be shifting focus to understand what are some of the internal use cases, especially from a DevOps and SRE perspective that you can drive with MongoDB Atlas. Uh, and the last section of this, uh, presentation will be, you know, we've been running a lot of hackathons across the country, so would love to walk you through some of the high level projects that were done at these DevOps hackathons. So, with that, let's get started with the session.

So, as we all know, you know, we've seen a lot of technology trends evolve over the years, right? Be it cloud computing, mobile internet, both at an enterprise level and a consumer level. All of these trends have had a high impact on how we live our lives.

Uh, but it took a couple of decades for a wide stream adoption. If you think about gen ai, it is one of the trends that has seen a massive impact and the fastest mainstream adoption, you know, a among all the trends that we've seen in the past. So certainly, you know, even at an individual level or at an enterprise level, each one of us has seen what an impact gen AI has made, and it has completely changed the way in we do things right?

It has changed the way we create new products, new content, new customer experiences. So it's certainly a game changer. So, with that effect, you know, Gartner had put across this estimate that 80% of enterprises will be using genai by 2026, and I'm sure that number is gonna be even higher, right?

Um, so knowing how critical, you know, gen AI and, um, gen AI is gonna be in the upcoming years, uh, we've built this maturity model so that we could share what we are seeing and hearing from our customers, right? So on the X axis, you'll see that it's the percentage of data that is operationalized as part of your Gen AI model. And on the y axis, you'll see what business impact it has, you know, what business impact your gen AI applications have in that case.

So we are seeing that a lot of, uh, organizations today are able to leverage the data, which is easily available. You know, and this data could be, uh, your structured data, right? The data, which is easily available, but the real differentiation, right?

And the way you will be able to kind of scale towards the top right is when you will be able to leverage unstructured data. And we know that 80% of, um, data today in organizations is unstructured, which means that if you're able to operationalize that unstructured data into your AI models, that's where you will be able to drive more differentiation with your AI strategy and with your AI models, right? And this could mean your free text, your images, your videos, um, you know, that's what we mean by unstructured data.

So as you look at this chart, think about where you are on this chart today, right? Um, what will it take for you to move to the top, right? And what are some of the hindrances that you're facing?

And the most important thing, what is the kind of data you're leveraging in the Gen AI models that you're building today? So before we go into some of the deep dive, I wanted to set some context around Vector, vector Search Rag, and the general Gen AI applications, right? So, as we know, gen AI applications leverage foundational models and embedding models, uh, to create, you know, the, the context, if you will, um, in terms of foundational models.

These are models that perform generative functions like creating your code, creating your text, et cetera. And some of the foundational models that you must have heard of are LLMs or large language models, right? Embedding models are similar to foundational models, uh, but they are the ones who take your unstructured data that we spoke about in the last slide and convert them into vectors, right?

Or into vector embeddings. What we mean by vectors are, it's a, it's a numeric representation of unstructured data using certain criteria, right? And it is these vectors that enable your gen AI applications to capture semantic meaning, um, and understand how similar or dissimilar to things or objects are, um, on, depending on how close they are in that space.

For example, you know, how similar is a cat to and versus a dog. That's just a simple example to share. Um, so again, you know, to kind of reiterate, vectorization is something which is critical to give context to gen AI models.

And one of the ways to feed these, uh, foundational and, uh, embedding models is through this architectural pattern called rag. So rag, which is also termed as retrieval. Augmented generation is an architectural pattern that enables you to deliver these gen AI powered applications, which are up to date with the latest and greatest information that are context aware, and it completely relies on your vector data.

So, going a level deeper into this, um, to reiterate, rag or retrieval, augmented generation relies on the ability to search vectors, right? We, we define what vectors are, or let's look at what this means. Um, so generally, if you look at your, uh, basic, um, you know, foundational models, uh, with rack businesses can take the reasoning capabilities from a pre-trained, pre-trained general purpose LLMs.

And once they are fed with, uh, proprietary information, right? The more value add comes from adding the proprietary data. This could be, um, you know, depending on the application you're working on, this could be company specific information or your customer information, or some billing information or the history, et cetera.

So when you are able to embed this proprietary data onto your foundational model, that's where, you know, you can build a well-trained LLM model, which is further refined with the different types of multimodal data, right? So this is where, um, you can, as compared to your competitors, provide the differentiated value. The more you can absorb, the more context rich data you can feed your foundational, uh, model with.

So vector search enables, uh, retrieving this most relevant data within the wide spectrum of data that you have at hand, right? So in this slide as an example, just to show you how vector search kind of does the magic, right? So for example, if you have a question which says, uh, compare and contrast the my retail expansion strategy to, uh, similar businesses in the space.

So this question is converted into, um, the numeric vector array that I had spoken to you about. So that's the way it is represented in space. And, uh, in order to retrieve the relevant context, and to answer that question, obviously the model needs to look at some documents or some pieces of information.

This could be your strategy docs, your industry reports, et cetera, which are also stored as vectors, uh, or vector arrays in the, uh, vector database, right? And with this context and with this vectorization that happens, and with the question that was, uh, uh, you know, um, initially po uh, raised, uh, this context rich, uh, information that exists along with vector search will allow your existing foundational model and give it the context richness that it needs, uh, to give you that differentiated and the context of your answer. So that's why vector search kind of plays a very important role when you are, uh, driving these important, uh, conversations.

And when you're building these gen AI powered applications. Now, you'll, you must be wondering like, how do I, I already have my database stack, right? I already use it.

So what are the approaches? How do I ensure that I can have vector, uh, and a vector database to kind of solve, uh, for these issues? Right?

So there are two approaches to do this. Uh, the first one is more of a bolt-on approach, where you can add, uh, a separate vector database, right? Let's, so we call it as a Bolton, it's a separate vector database on top of your operational database.

Now, this approach is fine, uh, but I wanted to highlight some of the challenges with this approach, right? The first one being that, uh, you'll, you are now looking at an additional database to manage, right? You might have issues with, um, querying it different to query languages.

You might want to keep the data in sync, uh, across these two databases or these two systems, which could be a manual process, um, from a developer perspective that could lead to a very fragmented experience, you know, when working across the two databases. And all in all, this would also lead to higher operational costs and a slower time to production. When you look at the second approach, uh, at the bottom, that's what we mean by a native database.

So, you know, leveraging your existing operational database with the vector capability. So basically you can utilize the same database to store your operational data as also your vector data and search, you know, using a single endpoint. Um, so here, obviously the advantage is that you have the single database, you don't have any additional systems to manage.

The data is automatically in sync, so you don't need to worry about that aspect. Um, and it's a much more unified way to work across your data, right? Um, which leads to lower operational costs, a better developer experience, and faster time to production.

So with that context, you know, our recommendation is looking at options where you have this built-in native Vector capabilities as part of your operational database. So with that context, I want to introduce you to MongoDB Atlas, uh, for gen AI powered applications. Um, MongoDB Atlas is a fully managed database offering from MongoDB, and as you can see, it certainly helps you unify your operational and vector database services all in the single platform.

As part of this chart, you'll see that vector search that we spoke about in the last section is a key part of this platform. And along with it, MongoDB also combines, uh, your operational database with full text and vector search, but also with additional features and functionalities such as, uh, stream processing time series, and more so with Atlas. Another important point I wanted to highlight is that it is a document.

We work with a document model and which means it's a much more intuitive way. This is something which our developers love, right? It's a much more intuitive way to work with your data.

It is, uh, all the information that is accessed together, stored together. They're already used to working with JSON. Um, so it's much more easier and intuitive for them to work with a document model.

And also, you know, you're using or you're working with a single consistent developer interface, um, with a single unified MongoDB query, API. So again, a much more rich developer experience to work with this model. Um, another important point I wanted to highlight here is that, uh, MongoDB Atlas is multi-cloud in nature, and, uh, we are the most widely available cloud-based developer data platform.

It's available across 117 regions across all around the world, uh, across the three major cloud providers, AWS Google Cloud, and Microsoft Azure. So certainly very highly available, um, supports multi-cloud, uh, capabilities and has a very resilient performance and secure platform, right? So, uh, certainly, you know, uh, we recommend MongoDB Atlas becomes very critical when you're building your Gen AI application.

So this was the overview, but I also wanted to go a level deeper in helping you understand how this becomes very relevant and how MongoDB Atlas becomes very relevant when you're looking at your Gen AI applications. Um, so the first thing I wanted to highlight here, and that I've spoken about a little bit on the last few slides is the criticality of having that single database, right, which stores your operational data, which stores your vector, uh, uh, you know, vector embeddings, and you're able to search and, uh, use a single endpoint to kind of work across these different types of data, you know, for your rich, uh, gen AI application. So that's what MongoDB uh, offers.

And again, you know, when we spoke about the maturity model in the second slide, this is what we are talking about, bringing in those rich capabilities, right? And being able to use it in a single platform. Um, the second point I wanted to highlight is the fact that you are able to also build richer functionality with MongoDB Atlas, right?

Uh, and very easily. For example, you are able to combine your vector search with other types of query, right? As an example, you might combine it with a geospatial query, and you might be able to ask a question around, um, recommend restaurants, which, you know, match my taste and which are in the 10 mile radius, right?

So here's where you are combining your vector search with your geospatial, um, uh, search query, right? So this again, makes for a much richer experiences and which makes it very, very suitable for your Gen AI applications. The third point I wanted to highlight is, uh, the document model.

So we kind of spoke about the document model a little bit on the last slide, especially when it comes to gen AI applications. The document model becomes very powerful, uh, when combined with vectors, because it allows for an easy model. Uh, it makes it very easy for you to model relationships between, uh, you know, your, uh, vectors and, uh, your metadata in a way that other databases cannot.

You know, for example, say you are putting together a database with a store, right? And you want information on the different books and the chapters within that books, um, you know, you could represent the chapters as vector arrays like we'd spoken about in the previous example. And all of this, you know, all the information about the book along with this chapters in a vector array are stored in a single document.

So this makes it much more easier than splitting up the data across different tables and then joining them together. It's a much more organized, um, this information is much more organized in a single document like I'd mentioned. So because of this, it makes it very, very easy to query and analyze the data in a much simpler and faster manner.

And again, you don't need to worry about complicated joints or any kind of post-processing steps that you would need, uh, with the traditional databases. So all in all, you know, this is a big value add that I wanted to highlight. The next thing I wanted to highlight was that it also enables you to have a cloud agnostic growth.

What I mean by that is, you know, uh, you might have your database deployed as an example on AWS, right? And as we all know, a lot of the cloud providers have these specialized AI services, if you will, right? Or AI infrastructure that is suitable for specific, uh, aspects of your application, and you might want to leverage that.

Now, the beauty of MongoDB is that it supports, um, you know, the, the multi cluster or the multi-cloud cluster support that we have, it allows you, you don't need to worry about actually migrating your infrastructure to GCP if you wanted to leverage an AI service there, you can easily expand your MongoDB Atlas cluster by creating additional nodes in GCP, which then allows you to leverage the specialized AI services that you want from GCP and gets you closer, you know, to, um, uh, to the GCP services. So even there won't be any performance or latency issues, right? So again, the whole aspect of, you know, having gen AI applications, the multi-cloud nature of these applications is what, uh, MongoDB supports with Atlas.

Um, and it's again, a very big value add when it comes to these use cases. Um, and the last point that I wanted to highlight was around, uh, just the rich ecosystem that we are part of, right? So we have, um, a deep integration with popular AI application frameworks, you know, be it Lang chain LAMA Index, um, all of the, uh, most of the application frameworks that you've heard of.

And also we support, uh, embeddings generated by the vast majority of model providers, right? So a lot of the foundational and embedding model providers such as open ai, hugging face, coherent, et cetera, all of these are supported, uh, by us, right? And it's a rich ecosystem of partners with whom we are consistently working, um, to give you that best experience you need when you are going to be deploying your Gen AI applications.

So with that, let's kind of change gears a little bit and talk about, um, how, or what are some of the use cases that you can think about, you know, using Vector search, uh, as it pertains to your DevOps and SRE use cases. We did speak a lot about, you know, the customer facing application, super critical. Uh, when you look at specifically DevOps and SRE, we all know that there are functions that are use cases that can certainly be optimized, and these could be your internal use cases that could be, uh, certainly optimized with the help of Genai models.

So I wanted to share a few examples here, right? The first example I wanted to share was around, uh, the optimized CI ICD pipelines, right? With Gen AI and MongoDB Atlas.

So utilizing vector search and the AI models that we spoke about, right? Um, you can enhance your decision making and operational efficiency by querying applications built on these lms, um, to ask questions around your CICD workflows and deployments. So as an example, you could include real time and historical data.

Uh, the, these could be your deployment records, these could be your performance metrics. Uh, you could ingest it through tools such as GitHub, uh, Jenkins, CircleCI, et cetera. The application framework then takes this information and is able to process this information, understand what are the workflow patterns, what are the terminologies that are unique to your organization, right?

Um, the ven vector embeddings are then stored in, um, MongoDB vector search. And then using your natural language, it becomes much more easier for you to ask questions around. Um, as an example, you know, you could ask a question around what are some of the common issues in the first five, um, or in the last five failed deployments, right?

That could be a question that you can ask, uh, or any other questions which can help you get a more understanding about your Dora metrics, right? These could be things around, uh, your change failure rate, your mean time to recovery. So all of this is possible by bringing together that rich data and using vector search.

Another example I would love to share is around incident management. Of course, this is a very critical use case when it comes to the SRE teams, right? And, uh, whenever you have any critical incidents, uh, being able to pinpoint, uh, the issues, right?

And being able to resolve it as soon as you can is key, right? But again, the data that is needed to resolve this can be so disparate and can be very difficult to kind of bring together for context, uh, rich, um, solutions, right? So this is what, uh, an example where you could use vector search and your AI model to kind of put this together.

So when it comes to incident management, you know, you could look at ingesting information like chat logs from Slack, you could use at, uh, you could use, uh, information like, uh, user activities, service tickets, um, error reports, et cetera. And you could ingested using tools such as PagerDuty, ServiceNow, Zendesk. So capture all of that information.

You could then use the application framework to add more, um, understand the workflow patterns, the terminologies, then create the vector embeddings, feed it into vector search, and then ask very specific questions around, uh, these incidents, right? And by combining all of this information, and with all of the, um, context, you can have much more pinpointed solutions and answers. Um, uh, in terms of like troubleshooting these incidents.

Um, these predictive alerts, you know, from your model can also then be channeled into tools that you use, such as PagerDuty or any other observability tools. And the last use case that I want to highlight here is around, um, auditing, again, a very critical use case. And, uh, with, with information that could be needed, you know, it could span across the different systems.

So it becomes very important to have a very targeted and context rich solution here. Um, so when you're looking at auditing teams or your DevSecOps teams trying to query and understand, um, any, uh, anything around the key audit incidents or any trends, uh, this solution could come in handy, right? You could look at ingesting security metrics.

These could be your security events, uh, severity of vulnerabilities, um, change management metrics, et cetera, from tools such as Splunk, uh, Qualys, SonarCube, Artifactory, et cetera. And again, with the similar approach that we spoke about earlier, use this information. Uh, you know, you could use the AI models to understand any patterns or terminologies that are specific to your organization, store these vector embeddings, um, and then use vector search to kind of ask, um, questions in a natural language.

You know, an auditor could simply ask questions like, what are some of the common vulnerabilities identified in the last quarter? Or in terms of trends they could ask, uh, show me some of the trends in change management incidents over the past year, right? So very specific questions around your, um, you know, your, your, um, audit incidents, trends, et cetera, can be asked used using this model.

So those were some examples I wanted to share, um, at a high level that you can implement for your internal efficiency across your DevOps and SR use cases. Uh, we've also been running a couple of hackathons across the country and specifically around, uh, the DevOps and SRE teams, right? So, uh, I wanted to highlight some projects or some, uh, key observations that I thought were very interesting and would be relevant for the audience here.

The first project, or one of the project that I thought was very interesting was around, um, producing code using an LLM model, right? So, uh, many of us must be aware of a, a practice that is used in software development today, which is called as test-driven development. So again, it is a development approach where you write tests for new features before you even write the code that make those features work.

So inspired by this practice or this test-driven development practice, um, you know, the team was able to provide test codes that outlined the necessary functionalities for the code that it needed to generate. So they were able to feed this to the LLM model. Um, this established the primary requirements for the task, right?

The LLM model then created a code or created the initial code based on these requirements. And this was the baseline, you know, for further improvement. So this, this was an iterative process where the LLM kept generating the code, and it was tested against, uh, the initial requirements.

And any failures were analyzed if there were any, uh, failures, the feedback was provided back to the LLM model. So as using the ative process, finally, uh, the code was able to successfully pass all the tests, and once this was achieved, the fully functional code was submitted. And this kind of marked the successful completion of the task, um, as per the test driven development framework.

So again, being able to use that similar approach, uh, to create this code, or asking the LLM to create this code using the TDD approach as something which was super interesting. Another example I want to share is, uh, how one of the teams was able to, um, you know, enhance the L chain's open AI agent. So what they did was, you know, um, in this example, the team implemented a natural language querying system for fast API endpoints using the open API agent.

Um, this enhancement aimed at improving the endpoint accuracy through a similarity search, allowing the system to better understand and process human language commands. Uh, and this was in context of a tool called Blender, which creates 3D models. So this team introduced, uh, a new similarity search feature to more accurately interpret any user commands that were, were given and match them with the appropriate API endpoint.

So, uh, even, uh, when the exact terms were not used, you know, so for example, substituting sphere for a circle, and all in all, this allowed or this helped minimize the interpretive, um, uh, the way the commands were interpreted by the, by the tool and thereby increase the tool reliability. The third use case that I want to highlight is, um, you know, one of the team helped kind of put together this use case around automating cloud infrastructure deployment using infrastructure code. So we are all aware of like infrastructure code, like cloud formation, Terraform.

In this example, they kind of worked with, uh, cloud formation and the model aimed to automate the deployment of AWS resources by utilizing an AI driven approach, right? So, um, the AI model was able to look at their current stack and suggest improvements, um, and basically deploy the resources or, you know, deploy the resources by getting a lot of this context. As an example, you know, the user might input, uh, in, in this example, the user added a command, like create an EC2 instance for me.

Uh, and a AI along with MongoDB Vector search was able to kind of look at the current state, right? Or was able to look at other information like documentation, uh, some GitHub examples, et cetera, to generate and execute the appropriate cloud formation commands. If the deployment, uh, was successful, it was reported back to the user if there was any error, uh, it indicated failure, right?

So, uh, the entire architecture helped, uh, you know, developers to deploy your cloud formation resources using, uh, AI model with a little bit of more context, um, and other resources. And the last example that I want to highlight here is around enhancing some of the AI powered coding agents. Um, so we are familiar with a lot of these AI powered coding agents such as Open devvin Auto Code Rover, which basically have transformed how software development, um, you know, how we are doing things with software development, right?

Be it generating code, identifying bugs. So these are the AI powered agents that are used today. Um, so in this use case, you know, the integration of MongoDB Vector search, uh, enhance these agents by enabling them to, uh, efficiently search and analyze, um, any of the commands, you know, or, uh, the keyword based searches.

So, you know, as an instance, or as an example, let's look at how this model work, right? So if someone added a question, right? How do, how to add elements, uh, in a list in Python?

So if you look at a traditional search, it might focus on within this question, it might focus on keywords such as add list, and that could result very generic results. You know, it, it doesn't really give the context rich information that you need. Uh, however, you know, with this team using Semantic Search and using Vector search, uh, they were able to understand the Query's context and retrieve the most relevant code snippets, uh, to answer this question, right?

So it was much more of a relevant way, uh, to look at the ask versus a regular search query, right? So this again, was a very interesting, uh, way to kind of enhance these AI powered coding agents that are used widely today. So those are a few examples that I wanted to share.

Um, and, uh, with this, I would like to leave you with a few resources that you can use to get started with Vector Search, with genai. Uh, the first one, uh, on the slide here is more of a program that we announced and we are super excited about this MAP program or the MongoDB AI applications program, which was launched just a few days ago here. And this brings together, you know, it gives you, uh, a straightforward way to get started with your Gen A application.

So it is executed, or you can work with a professional services to kind of get started with this, but it brings together, um, a lot of usage patterns that we've seen, right? Some of the common AI use cases, reference architectures, uh, we work with our partners and and such and bring you or help you to get started with your Journey AI journey. So feel free to take a look at that page.

Uh, I've also added a few pages on Vector search if you wanted to get started with, uh, an understanding of, um, our, our, um, take on ai. You know, I've added those web pages there. And then we also have some rich documentation around Vector search if you wanted to, you know, dive deeper into it.

Uh, we have a webinar which focuses on use cases, and the next few links are some free courses that you can, uh, access through MongoDB University to get started with Vector Search and ai. Uh, the last link is super, it's a blog, it's a tutorial, and I think it would be very relevant for this audience because as you know, you can get started with Vector Search using our Atlas ui, but you can also now use some of the tools that you're already familiar with, right? Many of you use infrastructure code tools such as Terraform Cloud Formation.

This tutorial mainly focuses on how you can start deploying and getting started with Vector Search using our MongoDB Terraform provider, right? So, uh, feel free to take a look at that and get started, uh, uh, with our Terraform provider, um, in implementing Vector Search. So that was all I had to share on the session today.

Uh, I hope this was valuable, and feel free to reach out to me. I've added my LinkedIn profile here. Feel free to reach out to me with any questions.

Um, thank you so much.