David Carter, Sapphire Ventures | DevOps Experience 2022

So thanks for having me. My name is David Carter and I head up the infrastructure in devops Center of Excellence at Sapphire Ventures. Sapphire is a grow stage VC focused on the Enterprise B2B sass ecosystem.

We spend a lot of time kind of investing and researching and diligencing the devops market in particular. I head up Sapphire's infrastructure and devops Center of Excellence, which is a dedicated function that supports the tech leaders in our portfolio with with strategy guidance and partner connectivity and peer-to-peer Community. My topic for today is all about autonomous devops and my plan is to share sort of State of the Union of the emerging platforms and projects and companies that are really defining this notion of autonomous devops.

And just forewarning before we begin. I went off the deep end a little bit for this talk with with AI generated images. There's there's one on this page from Dolly too stylized a lot of the presentation using images generated from stable diffusion and mid journey and others part of the reason I did this is just because I'm a geek and I think these are really cool really interesting Graphics, but also because the ml that underpins these tools things like gpt3 which we'll talk about in a bit are having a really major influence on the autonomous devops space.

So just sort of some method to the madness there. Today we're going to cover a few things. So we're going to cover off starting with an intro to the concept and just establishing some of the key terms but we mean when we say autonomous devops then we're going to talk about some of the ingredients that are feeling the shift that I'm going to cover some of the more interesting and sort of Novel capabilities that are coming to market across each phase of the sdlc and then we're going to wrap with some current state challenges and some of the kind of future implications of the advancements that are being made across this space.

So what do we mean by autonomous devops? So autonomous times describes an approach to software delivery in which applications are built deployed. And continuously maintained with little to no human input or intervention.

I put little two in in parentheses here because you know, really the purest definition of autonomous would imply that there's no human intervention. However, I see this road is sort of a journey in a way and and many aspects of autonomous devops exist today. They're just not fully end to end yet.

And so, you know, it's it's sort of a journey and and we're on our way but but not quite there fully a reasonable analogy to this would be sort of the state of self-driving cars, right? There's there's a lot we can do there but there are times when you definitely want to have your hands on the wheel. I put this word cloud at the bottom here.

These are all terms that you may have heard of that can be used to describe this General Trend. So I just added them here really just a level set and kind of keep us on the same page of what we're talking about today. So what is new here?

What is feeling the sort of evolution to fully autonomous devops? Well, there's a few market conditions that come to mind. The first is around environment complexity.

Right? So simply put you know developers need to know a lot of things these days. They need to know how to scale how to orchestrate how to add a debug they need to do this across many languages many run times architectures multiple Cloud providers doing all that while understanding how to defend against really sort of this ever evolving attack surface and orchestrating all that requires an understanding of different tools different utilities.

So I think you can see where I'm going with this specifies to say, there's just the set of skills and an effective developer most comprehend these days is pretty overwhelming and the goal post is just constantly moving right. There's always new tools to evaluate an old one to deprecate in the seemingly sort of endless cycle. The second condition here is Around Talent and specifically Talent constraints.

So people are burning out, you know, they're burning out due to some of the complexity that I just mentioned there's influence through inflation and wage stagnation, which really isn't helping things and you know, this is juxtaposed to the demand for developers, which is really never been higher The Wall Street Journal reported 340,000 new technical job postings. Just this past January which was up 11% year over year and this increased demand is sort of inhibited by the burnout, right which is still very high. 1 million people quit their jobs just this past September and a recent fuel stop study suggests that you know, 90 plus percent of majority of cios say that the sort of trend of the great resignation is still drastically impairing their ability to hire and retain technical Talent.

The third trend is really one of technical achievement. So we've seen really a step change in the efficiency and power of machine learning models and just the last few years. This is driven in part by Hardware improvements and just the availability of sort of massive amounts of compute from cloud providers.

But perhaps most importantly just significant advancements to machine learning and specifically like neural net architectures. The attention is all you need paper in 2017 really started with sort of most recent wave where the proposal of the Transformer which allowed for more efficient training and greater parallelism. And that was the basis for openai's GPT and it just feels like every day there's a new kind of academic paper that's coming out where someone's figured out a new sort of Novel optimization approach or way to squeeze more efficiency out of these models.

And so these advancements and AI ml are now being applied, you know, really across every facet of the devops life cycle. And me and a couple of my colleagues at Sapphire recently pulled together this landscape this autonomous devops landscape to help track some of the players that are kind of emerging across this ecosystem. The criteria that we used for this was to essentially highlight any company that's leveraging ml to power some aspect of the devops experience.

The list is you know, it's it's probably not comprehensive. I'm sure we missed some folks here, but it's an overall pretty solid representation of the market and the types of functionality that we see in scope here. A couple things to call out the first is that testing an AI testing an AI?

Ops so AI for it Ops in particular. Have some of the most vendors or entrants in this space. Those are two domains that have you know for lack of order terms sort of been at this the longest and really at the Forefront of applying ml to to the demops discipline.

We see lots of new players here on this slide, but also lots of incumbents many of which are starting to sort of acquire their way in so, you know tricentis for example acquired test them which is an AI testing platform service now bottom Loop systems, which is an AI Ops Focus tool and there's many many other applications examples of that. One other thing to note here is is just what's missing specifically we've omitted machine learning tool chain capabilities. So things like data labeling feature stores data quality tools model deployment model monitoring Etc.

I call it up because we see those as really foundational to the advancement of ml in general and super important to the advancement of atoms devops obviously, but just not particularly unique to devops specific use cases. So we so we didn't include those and sort of the scope of this landscape. um and look I know this has a bit of an eye chart.

So what I'm gonna do now is sort of drill down into each of these phases and highlight some of the more interesting and novel capabilities that are emerging across each of them. So starting with a design and plan phase, you know, we have this quote from The Meta AI team at the bottom, which I just love it says we believe that programmers should become a programming students should become a semi-automated task and which humans Express higher levels ideas and detailed implementation is done by the computers themselves. I just love this quote.

It's it's actually several years old now satis Chandra and team at metair just doing some really amazing research around the space. If you haven't already read I would I would recommend checking out there their papers on neural code search and API recommendations and and on utility fixes with get effects. It's just really interesting work from that team and I think this quote, you know, the vision for it really starts at the design phase where you begin to express some of these higher level ideas.

And so a few interesting capabilities that are emerging here. The first is around requirements analysis. So companies like scope Master are using NLP to review requirements and user stories for duplicates for omissions for inconsistencies.

And then to essentially recommend and make changes to these to make them clear and even go to the next step of providing delivery estimates based on the scope of the work to help sort of automate and and optimize the requirements phase at the design phase or designing phase. You've got computer vision tools that are being used to transform a hand-drawn sketch into a design time artifact. So something like a figma file and then some really taking those files and actually being able to convert them into Deployable front end application code all just essentially with a click of a button.

And then finally the auto refactoring space is super interesting. You've got monoliths which are you know silvery common particularly with Brownfield applications particularly within you know, sort of Legacy Traditional Enterprises larger Enterprises that have been around for a while. Um teams look they often Endeavor to refactor monolith and to discrete microservices, but the process is complex and it takes a bunch of time and there's lots of proceed risks and so usually get sort of puts aside or delayed and emerging companies like V function are aiming to sort of flip this on its head and really automate this process.

So they're in particular using machine learning to scan source code and identifying and recommend sort of natural break points with which the application can be decomposed and then actually automating the necessary code changes to break down monoliths into microservices. Another interesting tool here is modern. They have a project called open rewrite where they're automating dependency updates.

So they use semantic code search to spot potential issues that might come from upgrading and outdated or vulnerable package and then they make the necessary code changes to facilitate that update without breaking a build. So really interesting Trends here. At the code and build phase.

This is arguably the hottest or kind of most trending aspect of autonomous devops. The code and build phase is where we're seeing these generative coding platforms like tab 9 and replen and co-pilot start to emerge. These tools are capable of recommending everything from single line autocomplete to full functions.

And they can also do things like convert from one language to another they can Auto add comments and documentation and they do this all in response to user provided problems. And you know the neural Nets that are powering these things they've been around for some time, right the language models the neural Nets but there's been a recent push towards applying those sort of foundational Technologies towards code and the commercialization of those efforts is still just sort of an incredibly new phenomenon. Um, so just a quick I just want to quickly actually touch on this and some of the history here.

I mentioned earlier the Advent of the Transformer and now open AI use that to architect and build GPT. Well, we can kind of fast forward to 2019 when Microsoft invested a billion dollars into open AI in Exchange open AI essentially exclusively licensed a bunch of their Tech to Microsoft including gpt3 but codex was then build on top of gpt3 and that was really the industry's first multi-billion parameter language. So large language model Focus specifically on code.

Codex gets a lot of attention because of co-pilot but there are other interesting Alternatives that have emerged things like Salesforce is codegen which is built on top of T5 that came out of Google. Most of these models are trained on English language web crawl data, and then they further refine them on top of code bases. So in the case of codecs they use Python code.

Codex or codegen sorry took a similar approach using web crawl data, but then calibrated on six different code bases. So kind of mixed in a polygon sort of language Corpus. Tools like replicate copilot tab 9 are again a really at the Forefront of the application of these models towards devops and these guys, you know, they're measuring Effectiveness and value in terms of accepted recommendations or ghost code except it goes code in addition to other sort of standard benchmarking Frameworks.

So co-pilot was just G8 this past June and just a short, you know few months is already cleaning some pretty eye-popping statistics. They make that on average users are accepting 26% of all the recommendations coming out of their tool and on average 27% and developers files were generated by co-pilot for the projects where it's being applied and in the case of certain models, like python, you know, those numbers can go as high as 40% python bit again ties back to the influence of training data and the importance of training data because codex was again trained on Python, and so it tends to perform better with that particular language. Um, look, I some of the stuff is difficult to measure and you know, you could say that the code was accepted the recommended recommended code is accepted but then the tools kind of lose sight and whether it was significantly edited after it was accepted so they may not have visibility to that.

But either way the numbers are incredibly impressive some of the sort of patterns of utilization we've seen people reporting that these tools are really additive and helpful with doing things like trivial copy and pasting of scaffolding code where there's a very sort of clear pattern. That's that's emerging. You can just sort of like copy down as well as things like suggesting API methods if you're starting to interact with a new API and maybe you know, you're not as familiar with it.

They can sort of help you navigate that another thing we've heard and sort of seen is more senior developers tend to stick to single line auto complete versus the full function recommendations, which is has more to do I think with you know, their understanding and familiarity with a code base and so for them, it's really just about sort of the efficiency of the single line sort of like completing a sentence if you will. While these tools are super effective and promising against statistics are pretty eye-popping really they should be checked by Engineers before being sort of blindly accepted right the recommendations, you know, there's not a lot of sort of like post-check validation. I'll talk about this in a minute.

But again, it goes back to the self-driving car analogy. Right? Like we still need our hands on the wheels here to some extent.

At the scan and testing phase, you know developers write multitude of tests. Just part of sort of be Au activities things performance test integration tests Etc super critical, but also incredibly cumbersome to write to maintain over time as code changes smarter conducted a survey this year in which nearly half of the engineers. They pulled reported spending more than 70% of their time.

Just writing and executing tests. And test Suite tends to tend to balloon over time, right? So Shopify for example has talked about their their test suite for their monolith.

Their core monolith is comprised of 170,000 tests. So lots of opportunity here to sort of optimize. Testing in particular like autonomous devops is a super good.

It's super important to this notion of autonomous devops because of what we just talked about right the code generation tools. They're great, but they're not particularly good and validating the code, they generate and so again testing plays the sort of critical role as sort of a tollgator if you will. Some functionality here that's interesting synthetic generation tools like tonic and AI are using machine learning to Auto auto generate test data.

So teams get more sort of effectively stimulate production workloads without eating too expose sensitive data to lower environments. There's auto test generation tools companies like relics eggplant prod perfect that are actually analyzing application traffic. So it's like functional sort of traffic within the application functional sort of transactional traffic to inform which test they write so they aim to write tests that are based on sort of flows within the application that are most commonly access by end users.

Circle CI Pony code and diff blue or two sort of interesting players in the unit test generation space. So unit tests are again. Just huge time suck very tedious to write and maintain the blues using reinforcement learning which is capable of validating that the test work before they sort of publish them which again is moving a sort of much closer to sort of fully autonomous session.

That's prioritization. The aim here is to you know, reduce computational load on test runs and improve sort of the speed of test run. So the idea is to select tests and order them in a way where they're covering the code that's actually impacted by a given change and a sort of prioritize those first.

So companies like see lights as an example are interesting players here visual QA tools are using Vision AI to sort of analyze them to an individual pixel level the consistency and sort of like intended functionality of a gaming application across different devices and browsers and platforms as well as to do things like validated here into Ada and disability standards accessibility standards. Semantic colon scanning tools. So from companies like deep code which sneak Acquired and sell which which the GitHub required and actually form sort of the co-pilot team.

They're using ml driven semantic code analysis to speed up scan times and reduce false positives. And then the last one here to highlight is self-healing tests and self-healing code and the idea is using elements within a given test script and automatically updating those elements as they're of changes observed to the code. So for example, the name of a Dom element changes to go in and update that functional test.

So that the test doesn't break the next time it runs this also applies to scanning. So companies like americode with their juruna acquisition have introduced Auto remediation capabilities, which is done in part through machine learning analysis of past vulnerability fixes. So when they identify a vulnerability a scan actually able to go in and recommend and automate the fix based on observation of sort of like past ways that that fix has been handled so moving to the release phase.

This is where we're trying to optimize for time to feed back and this is really being driven by this notion of Shifting right and testing in production. You know machine learning tools have really begun to unlock the shift right Paradigm and and make it sort of more feasible and sort of address a lot of the concerns that have historically existed around shifting right and testing and prod and introducing buggy buggy code to customers and things like that. So one capability to highlight here is this notion of change risk prediction prediction digital AI bought a company called numerify which has some really compelling features here.

The idea is that their platform assesses test code coverage. They assess dependency of a Target system. They assess the past results of similar code changes the past performance of the developer that's contributing the code they wrap all that up and and sort of use it to drive and Define a dynamic set of toll Gates where each release depending on sort of the risk that they quantify may have a different set of toll Gates that they that they determine they might decide for example to change like the time window that the release is done.

They might decide to in to require some additional code reviewers and other steps that are just meant to mitigate some of the risk for given change. Continuous verification is another really interesting concept the platform here the platforms here coupling the automation sort of multi-cloud deploy capabilities of CD platforms with the Telemetry and sort of like robot robust analytics of modern observability platform. So whereas a team in the past may have performed some manual checks post-deployed to determine the success of the release continuous verification platforms do all this for you.

So they ingest the data they from a multitude of sources they apply ml to assess sort of performance against baselines Tech deviations, and then actually autonomously roll back applications in the event of a perceived issue. So companies like circling company called Vamp and harness a sort of leading the charge here. And then last one in this phase Auto tuning features.

So feature flagging experimentation, really hugely important aspect of Shifting right and some of the emerging platforms like static better players in the space are using machine learning to observe user behavior and auto present to them variants that optimize for a given desired metric. So if you're trying to optimize on something like number of clicks that they will sort of like use behavioral analysis to determine like okay this particular features sort of the best one of a choice of a choice of different experimentation options for you to sort of perform and execute on that desired metric and then monitoring maintaining, you know, I want to highlight. I don't want to spend a ton of time in this I think this space has been around for quite some time this idea of AI for it Ops or AI Ops the general capabilities here.

I've listed out things like noise reduction anomaly detection root cause analysis and then sort of Downstream automation. Like sort of executing run books. It's not to say that the space is an important.

It's just one that sort of more mature has been around for a while. There's there's like pure play vendors that have existed here like movesoft and big Panda and it starts there's more itsm-focused players like service now and pay for Duty and move works and then incumbents that you know are sort of extending their traditional observability platforms with machine learning. Companies like New Relic and Dyna Trace, but super important really really compelling stage.

Just you know it more mature. I would say arguably then a lot of the other phases that we've talked about here. So starting to wrap up here some of the challenges right?

There's a lot of interesting sort of excitement in this space. But what is the catch right? Like what are some of the challenges that the market is faced with which is preventing more sort of widespread adoption.

The first around training data quality. This is that old adage, you know garbage in garbage out these. Llm Transformers are very good at spotting patterns and making predictions based on user prompts and sometimes you know users will supply good prompts which which can lead to the models recommending good code and sort of quality code and sometimes users can provide incomplete or sort of bad prompts which can lead to you know, the opposite which is bad code or bad recommendations and the models, you know, they're not they don't care really like they're happy to generate either based on what they've been shown and so if you're coding say outside of your comfort zone like you're trying to use co-pilot to learn rust and you want to start building in a new language you might for example, like want to sanitize and SQL variable for a new language making SQL call.

If you don't explicitly ask the model. Hey, like I want to write this SQL call and the database call and actually sanitize my variables the model probably won't do that. Like if you don't prompt it properly it's just Going to take that step.

And so there's some challenges on the prompting side. But also on the quality of the code side, right the quality of the code. The models are given.

So GitHub is this massive repository. There's tons of old sort of Legacy code in it. There's poorly written code.

There's insecure code mixed in with all the really great examples of code and all of that gets baked into the model sort of the same right? It's also the treated the same so training data super important having sort of a large enough Corpus of data. That's that's quality data is very key.

Cyber, security exploits look researchers from NYU have recently published a study in which they claimed I think up to 40% of co-pilot recommendations included some type of vulnerability and it particularly struggled with with security sensitive code. So if you say wanted it to generate for you the login form on a web page it was it was not particularly good doing that in a way where it would provide sort of a secure answer to that question. The training data here again is a key sort of impact or influencer same with the prompts the researchers for example actually found that the strongest indicator of whether co-pilot would recommend voterable code is whether there was already sort of a vulnerability or multiple vulnerabilities within the code base of the of the user.

And again, that's part that gets sort of baked in the files that I have open get baked into the prompts. I think, you know this in particular like this sort of behavior kind of amplifies that challenge here when you think about it like someone with a bunch of insec Code in their projects is probably less likely or less inclined to notice and sort of like take account of the fact that the machine generated code is also insecure so it sort of compounds the issue in a way. There's also IP challenges and there's been you know, several sort of recently publicized examples of people observing co-pilot and claiming that it is essentially just sort of copy and pasting full Snippets of code without providing proper attribution proper sort of call out to the language or sorry the licenses.

Um, there's actually an active velocity right now against Microsoft which makes similar claims for what it's worth. You know. This is a really significant challenge GitHub claims that co-pilot is not intended to function this way.

I don't you know, personally think this is happening intentionally and maliciously, but at the same time I completely understand why people would be upset and why if they're work isn't getting sort of proper attribution. They would be upset about that. I expect GitHub to kind of fix this over time and likely by providing some type of Downstream IP checking functionality prior to sort of like posting the recommendations, but it's a concern for now, but I I expect over time, you know, the sort of Industry will figure out a way to sort of like address this Maintainability is another serious challenge the further and further removed developers are from writing code really the first further and further.

They may find themselves from understanding the code that's written which can have obvious ramifications down the line from a supportability perspective. If suddenly you've introduced a bug and it came off the back of some code that was recommended by an AI going into troubleshooting that and actually sort of fixing the code might be prohibitive. And then the last thing to call out is this idea of machine reasoning and so Look, the models are very good at predicting the next token, right?

But but they also, you know, they tend to struggle on lack kind of business contact. So they might not know kind of what the actual end-to-end desired state of application with the actual like intended and user functionality is supposed to be This makes it really difficult for the models then to validate whether the code they're generating is actually good or actually works and it really opens the door to potential sort of codification. If you will of erroneous Behavior or incorrect Behavior.

I think for now, you know, some of the ideas that have been proposed to here is coupling the models and I started hybrid manner with say like a rules-based semantic engine which After they recommend code will then go and sort of validate whether the code is is of a certain quality. But again that is really still a bit of a Band-Aid and and done within sort of the narrow context of the given recommend recommended code snippet versus having this broader understanding and sort of reasoning of the end to end application. So I want to wrap with some thoughts around the future implications of these Trends and I'm gonna do that through the lens of these three vectors the machine learning models the engineering experience and the vendor ecosystem on the machine learning models.

Look, I think one thing that's super important here is for the open source Community to really kind of contribute more and kind of way in here. They're still really isn't a great sort of like de facto open source standard or framework for code Generations in particular things like codex are proprietary. It's it's licensed by open Ai and they don't really give away all the secrets about how they be process the model how they optimize the model.

And I think having a viable open source, ecosystem will be really key here to sort of like driving and Associated explosion of new tools and vendors in these spaces. The good news is researchers have started to address this Gap and at places like Google and CMU. They're doing some really foundational work to push effective open source models out to the community.

So things like T5 and polycoder are really great examples of this some more to come there but looking forward to, you know, more sort of advancement and availability of really powerful models in the open source domain. Training data quality and privacy. I spoke about this.

So, you know, the general idea is we need to provide better data sets with with a stronger Corpus of positive and sort of properly secure code examples to reduce the potential for recommending buggy coder and secure code. But even beyond that, I think what we'll start to see is this trend where we'll have sort of a hybrid training model where the foundational models are. Then further refined on top of private data sets.

So things that are not available within a public repo. So this is say like the source code for a large bank is an example companies like tab 9 or essentially already doing this, right? They they take sort of publicly available data and they mix it with private data to provide sort of more tailored more focused examples of sort of like last code examples that again for privacy reasons, the company might not want to expose but that they they would like ultimately for the AI to recommend because it aligns with their coding style or their particular preferences for how soft written that they're particular company.

And then on efficient training look I love this Top This Time Goldstein tweet. I think it just helps crystallize really like what is physically involved in training like a half a trillion parameter model like Google's Tom and with every version of like a GPT the models just get larger with more parameters and require more and more compute which always just over time. Like it's it's really different to scale that and and there's there needs to be sort of optimizations.

The interesting thing is like, you know Google published palm and and that this is happening trillion parameter, but they also recently published a paper about this model. They build called chinchilla and which they were able to significantly optimize compute and query performance. Well, achieving, you know performance that's similar to like a gp3 using significantly less compute power.

So I think that the general sort of push here the idea here is like we need more like chinchillas, right? We need we need more promising models which can achieve the performance of a gp3 or a codex with with the more efficient. 0 many years ago, which which is was this idea of the sort of seismic shift where SCLC phases would be dissolved into this more sort of fluid pipeline where machine learning is used to to write the software itself and the role of developers sort of transitions from that of sort of makers into Checkers and sort of like steers of the AI.

To that end. There's there's a lot of Buzz around this idea of prompt engineering which is like this emerging discipline which would occur within Davos where Engineers would be tasked with with essentially wrangling the model. Right?

If you've ever interacted with Dolly 2 or stable diffusion or these AI image generation tools. I think you probably already have experience with this. It's like you you write your first prompt the sentence that you want.

It returns an image and your brain immediately flips into kind of prompt engineering mode where you think like, okay. What else? Can I write?

Like, how can I tweak my prompt in order to kind of bend the model in a way to get it to hang me back the image that I actually wanted? Ultimately, you know, I think these Trends will trigger the sort of further expansion of the skills that are required to Be an Effective devops engineer. So things like data engineering and machine learning disciplines, like data quality and model monitoring and things like that.

I think we'll start to become part of you know, what we consider to be a full stack devops engineer if you will. And then finally, you know, I think that the dev interface potentially is going to sort of shift here. So, you know, most of this is centered around the IDE today, but I think is as these new tools start to emerge.

It's gonna be really interesting to track like what the actual implications are on the developer interface. Like will this become more sort of a conversational thing that you that you perform and it's sort of a natural language with an AI. Will it become more sort of a sketch and design focused discipline?

And fundamentally, I think this is going to alter the sort of again the types of skills and backgrounds that make for an effective engineer. And then last thing to close with vendor ecosystem. So look, I think one really interesting trend is the intersection with low code.

Historically low code has focused the idea and premise of low code is focused on this democratization of application development, right and allowing people that are non-technical users to essentially build and construct applications. I think historically software developers were often quick to kind of dismiss local tools. Basically saying like they lack the sophistication they lack the ability to generate sort of like meaningful like useful code.

The irony here and I think the thing that's so interesting is you know, these low-code tools what we're talking about with generative coding tools it they are local tools. Like it's it's the same concept right? Like it is auto generating things off the back of sort of like natural language prompts.

It is a generative code tool like co-pilot is low code, it's just approaching it from a different angle, right the low code tools of old like more traditional platforms like out systems and mendix. They're using very highly opinionated sort of fixed templates versus Copa. There's using, you know, really sort of like Cutting Edge machine learning techniques at the end of the day.

It's they're all sort of like in sort of a general sort of concept low code. And I think what's interesting is the low code providers themselves are perhaps some of the best position to sort of provide the type of modeling and sort of design first interfaces that will be relevant in the future of the space. We've already begun to see that start to take shape.

Right? So like Microsoft is taking gpt3 and applied it to their power apps platform in order to enable non technical users to describe and build data queries using natural language and I expect that Trend to continue and it'll be interesting to see whether the low Co providers the more traditional ones again capture more of the sort of autonomous devops Market. One other thing we'll expect to see here is a true kind of end-to-end devops plot autonomous devops platform.

So, you know, we've shared a lot of sort of interesting sort of bespoke features throughout this presentation and interesting products, but there's a yet like a single platform that sort of pulls all of us together into sort of like a common autonomous devops. Form, so there isn't a tool that's taking sort of design and coding and testing deploy monitoring feedback Etc and pulling that all into like a single unified autonomous devops kind of sweet. but we expect that to happen and and so so more to come there but you know, we'll look at it over time and somebody is bound to emerge and I think the next one the next sort of atlassian or gitlab the next sort of major platform provider in the devops space will be really a team that sort of like expert it's a machine learning and compress the data scientists and really kind of on the Forefront of this autonomous trend And then finally, you know goes up thing.

But there's just massive opportunity here as these models get more powerful is they become more and more sort of accessible in the open source domain for emerging startups products, like co-pilot are really just sort of the tip of the iceberg and and we see, you know host of other opportunities where these large language models can be applied and you startups we believe we're gonna start to emerge so it's all wrap with that. I just, you know, I want to just say it's just it's it's so cool. It's something like exciting time to be sort of focused on this this ecosystem.

And, you know sort of super interested to see what the future of the space will hold if you're a company that's building functionality in a space. We'd love to talk to you. com.

And yeah, thanks for for watching and thanks again for for taking the time.